Welcome to ChubaoFS(Chubao File System)

Introduction

ChubaoFS(Chubao File System) is a distributed fle system that is designed to natively support large scale container platforms.

High Level Architecture

Architecture

ChubaoFS consists of a metadata subsystem, a data subsystem, and a resource manager, and can be accessed by different clients (as a set of application processes) hosted on the containers through different file system instances called volumes.

The metadata subsystem stores the file metadata, and consists of a set of meta nodes. Each meta node consists of a set of meta partitions.

The data subsystem stores the file contents, and consists of a set of data nodes. Each data node consists of a set of data partitions.

The volume is a logical concept in ChubaoFS and consists of one or multiple meta partitions and one or multiple data partitions. Each partition can only be assigned to a single volume. From a client’s perspective, the volume can be viewed as a file system instance that contains data accessible by the containers. A volume can be mounted to multiple containers so that files can be shared among different clients simultaneously, and needs to be created at the very beginning before the any file operation. A ChubaoFS cluster deployed at each data center can have hundreds of thousands of volumes, whose data sizes vary from a few gigabytes to several terabytes.

Generally speaking, the resource manager periodically communicates with the metadata subsystem and data subsystem to manage the meta nodes and data nodes, respectively. Each client periodically communicates with the resource manager to obtain the up-to-date view of the mounted volume. A file operation usually initiates the communications from the client to the corresponding meta node and data node directly, without the involvement of the resource manager. The updated view of the mounted volume, as well as the file metadata are usually cached at the client side to reduce the communication overhead.

Features

Scalable Meta Management

The metadata operations could make up as much as half of typical file system workloads. On our platform, this becomes even more important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system. The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have hundreds of billions of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting.

General-Purpose Storage Engine

To reduce the storage cost, many applications and services are served from the same shared storage infrastructure (aka “multi-tenancy”). The workloads of different applications and services are mixed together, where the file size can vary from a few kilobytes to hundreds of gigabytes, and the files can be written in a sequential or random fashion. For example, the log files usually need to be written sequentially in the execution order of the code; some data analytics in the machine learning domain are based on the data stored on the underlying file system; and a database engine running on top of the file system can modify the stored data frequently. A dedicated file system needs to be able to serve for all these different workloads with excellent performance.

Strong Replication Consistency

E-commence venders who move their line of business applications to the cloud usually prefer strong consistency. For example, an image processing service may not want to provide the customer with an outdated image that does not match the product description. This can be easily achieved if there is only one copy of the file. But to ensure a distributed file system to continue operating properly in the event of machines failures, which can be caused by various reasons such as faulty hard drives, bad motherboards, etc, there are usually multiple replicas of the same file. As a result, in a desired file system, the data read from any of the replicas must be consistent with each other.

Relaxed POSIX Semantics and Metadata Atomicity

In a POSIX-compliant distributed file system, the behavior of serving multiple processes on multiple client nodes should be the same as the behavior of a local file system serving multiple processes on a single node with direct attached storage. ChubaoFS provides POSIX-compliant APIs. However, the POSIX consistency semantics, as well as the atomicity requirement between the inode and dentry of the same file, have been carefully relaxed in order to better align with the needs of applications and to improve the system performance.

Run Cluster by Yum Tools

Yum tools to run a ChubaoFS cluster for CentOS 7+ is provided. The list of RPM packages dependencies can be installed with:

$ yum install http://storage.jd.com/chubaofsrpm/latest/cfs-install-latest-el7.x86_64.rpm
$ cd /cfs/install
$ tree -L 2
.
├── install_cfs.yml
├── install.sh
├── iplist
├── src
└── template
    ├── client.json.j2
    ├── create_vol.sh.j2
    ├── datanode.json.j2
    ├── grafana
    ├── master.json.j2
    └── metanode.json.j2

Set parameters of the ChubaoFS cluster in iplist.

  • [master], [datanode] , [metanode] , [monitor] , [client] modules define IP addresses of each role.
  • #datanode config module defines parameters of DataNodes. datanode_disks defines path and reserved space separated by “:”. The path is where the data store in, so make sure it exists and has at least 30GB of space; reserved space is the minimum free space(Bytes) reserved for the path.
  • [cfs:vars] module defines parameters for SSH connection. So make sure the port, username and password for SSH connection is unified before start.
  • #metanode config module defines parameters of MetaNodes. metanode_totalMem defines the maximum memory(Bytes) can be use by MetaNode process.
[master]
10.196.59.198
10.196.59.199
10.196.59.200
[datanode]
...
[cfs:vars]
ansible_ssh_port=22
ansible_ssh_user=root
ansible_ssh_pass="password"
...
#datanode config
...
datanode_disks =  '"/data0:10737418240","/data1:10737418240"'
...
#metanode config
...
metanode_totalMem = "28589934592"
...

For more configurations, please refer to Resource Manager (Master); Meta Subsystem; Data Subsystem; Client; Monitor.

Start the resources of ChubaoFS cluster with script install.sh . (make sure the Master is started first)

$ bash install.sh -h
Usage: install.sh [-r --role datanode or metanode or master or monitor or client or all ] [-v --version 1.5.1 or latest]
$ bash install.sh -r master
$ bash install.sh -r metanode
$ bash install.sh -r datanode
$ bash install.sh -r monitor
$ bash install.sh -r client

Check mount point at /cfs/mountpoint on client node defined in iplist .

Open http://consul.prometheus-cfs.local in browser for monitoring system(the IP of monitoring system is defined in iplist ).

Run Cluster Manually

Building

Use following command to build server and client:

make build

If the build is successful, cfs-server and cfs-client will be found in directory build/bin

Deployment

Start Resource Manager (Master)

nohup ./cfs-server -c master.json &

Sample master.json is shown as follows,

{
  "role": "master",
  "ip": "10.196.59.198",
  "listen": "17010",
  "prof":"17020",
  "id":"1",
  "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",
  "retainLogs":"2000",
  "logDir": "/cfs/master/log",
  "logLevel":"info",
  "walDir":"/cfs/master/data/wal",
  "storeDir":"/cfs/master/data/store",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "exporterPort": 9500,
  "clusterName":"chubaofs01",
  "metaNodeReservedMem": "1073741824"
}

For detailed explanations of master.json, please refer to Resource Manager (Master).

Start Metanode

nohup ./cfs-server -c meta.json &

Sample meta.json is shown as follows,

{
    "role": "metanode",
    "listen": "17210",
    "prof": "17220",
    "logLevel": "info",
    "metadataDir": "/cfs/metanode/data/meta",
    "logDir": "/cfs/metanode/log",
    "raftDir": "/cfs/metanode/data/raft",
    "raftHeartbeatPort": "17230",
    "raftReplicaPort": "17240",
    "totalMem":  "8589934592",
    "consulAddr": "http://consul.prometheus-cfs.local",
    "exporterPort": 9501,
    "masterAddr": [
        "10.196.59.198:17010",
        "10.196.59.199:17010",
        "10.196.59.200:17010"
    ]
}

For detailed explanations of meta.json, please refer to Meta Subsystem.

Start Datanode

  1. Prepare data directories

    Recommendation Using independent disks can reach better performance.

    Disk preparation

    1.1 Check available disks

    fdisk -l

    1.2 Build local Linux file system on the selected devices

    mkfs.xfs -f /dev/sdx

    1.3 Make mount point

    mkdir /data0

    1.4 Mount the device on mount point

    mount /dev/sdx /data0

  2. Start datanode

    nohup ./cfs-server -c datanode.json &

    Sample datanode.json is shown as follows,

    {
  "role": "datanode",
  "listen": "17310",
  "prof": "17320",
  "logDir": "/cfs/datanode/log",
  "raftDir": "/cfs/datanode/log",
  "logLevel": "info",
  "raftHeartbeat": "17330",
  "raftReplica": "17340",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "exporterPort": 9502,
  "masterAddr": [
     "10.196.59.198:17010",
     "10.196.59.199:17010",
     "10.196.59.200:17010"
  ],
  "disks": [
     "/data0:10737418240",
     "/data1:10737418240"
  ]
}

For detailed explanations of datanode.json, please refer to Data Subsystem.

Start ObjectNode

nohup ./cfs-server -c objectnode.json &

Sample objectnode.json is shown as follows,

{
    "role": "objectnode",
    "domains": [
        "object.cfs.local"
    ],
    "listen": 17410,
    "masterAddr": [
       "10.196.59.198:17010",
       "10.196.59.199:17010",
       "10.196.59.200:17010"
    ],
    "logLevel": "info",
    "logDir": "/cfs/Logs/objectnode"
}

For detailed explanations of meta.json, please refer to Object Subsystem (ObjectNode).

Create Volume

By default, there are only a few data partitions allocated upon volume creation, and will be dynamically expanded according to actual usage.

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=10000&owner=cfs"

For performance evaluation, extra data partitions shall be pre-created according to the amount of data nodes and disks to reach maximum performance.

curl -v "http://10.196.59.198:17010/dataPartition/create?name=test&count=120"

Mount Client

  1. Run modprobe fuse to insert FUSE kernel module.

  2. Run yum install -y fuse to install libfuse.

  3. Run cfs-client -c fuse.json to start a client daemon.

    Sample fuse.json is shown as follows,

    {
  "mountPoint": "/cfs/mountpoint",
  "volName": "ltptest",
  "owner": "ltptest",
  "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",
  "logDir": "/cfs/client/log",
  "profPort": "17510",
  "exporterPort": "9504",
  "logLevel": "info"
}

For detailed explanations of fuse.json, please refer to Client.

Note that end user can start more than one client on a single machine, as long as mountpoints are different.

Upgrading

  1. freeze the cluster
curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"
  1. upgrade each module
  2. closed freeze flag
curl -v "http://10.196.59.198:17010/cluster/freeze?enable=false"

Resource Manager (Master)

The resource manager manages the file system by processing different types of tasks, such as creating/deleting/updating/loading partitions and keeping track of the resource status (such as the memory/disk utilization). The resource manager is also responsible for creating new volumes and adding new meta/data nodes to the ChubaoFS cluster. It has multiple replicas, among which the consistency is maintained by a consensus algorithm such as Raft, and persisted to a key value store such as RocksDB for backup and recovery.

Utilization-Based Distribution/Placement

The resource manager is a utilization-based distribution strategy to places the file metadata and contents across different meta and data nodes. Because each node can have multiple partitions, and the partitions in a volume do not need to reside on the same node, this distribution can be controlled at a finer granularity, resulting a more efficient resource management. Specifically, the distribution of file metadata and contents works follows:

First, when mounting a volume, the client asks the resource manager for a set of available meta and data partitions. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Later on, when writing a file, the client can only choose the meta and data partitions in a random fashion from the ones allocated by the resource manager.

Second, when the resource manager finds that all the partitions in a volume is about to be full, it automatically adds a set of new partitions to this volume. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Note that, when a partition is full, or a threshold (i.e., the number of files on a meta partition or the number of extents on a data partition) is reached, no new data can be stored on this partition, although it can still be modified or deleted.

Replica Placement

When choosing partitions for the replicas, the resource manager ensures that two replicas of the same partition never reside on the same node.

Meta Partition Splitting

There is a special requirement when splitting a meta partition. In particular, if a meta partition is about to reach its upper limit of the number of stored inodes and dentries, a splitting task needs to be performed with the requirement to ensure that the inode ids stored at the newly created partition are unique from the ones stored at the original partition.

To meet this requirement, when splitting a meta partition, the resource manager cuts off the inode range of the meta partition in advance at a upper bound end, a value greater than highest inode id used so far (denoted as maxInodeID), and sends a split request to the meta node to (1) update the inode range from 1 to end for the original meta partition, and (2) create a new meta partition with the inode range from end + 1 to infinity for this volume. As a result, the inode range for these two meta partitions becomes [1, end] and [end + 1, infinity], respectively. If there is another file needs to be created, then its inode id will be chosen as maxInodeID + 1 in the original meta partition, or end + 1 in the newly created meta partition. The maxInodeID of each meta partition can be obtained by the periodical communication between the resource manager and the the meta nodes.

Exception Handling

When a request to a meta/data partition times out (e.g., due to network outage), the remaining replicas of this partition are marked as read-only. When a meta/data partition is no longer available (e.g., due to hardware failures), all the data on this partition will eventually be migrated to a new available partition manually. This unavailability is identified by the multiple failures reported by the node when operating the files.

Metadata Subsystem

The metadata operations could make up as much as half of typical file system workloads. this can be important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance/storage bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system.

Internal Structure

The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have a set of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries.

Generally speaking, An inode is an object that represents the underlying file (or directory), and can be identified by an unsigned 64-bit integer called the inode id. A dentry is an object that represents the directory hierarchy and can be identified by a string name and the id of the parent inode. For example, if we have two directories foo and bar, where foo is the parent directory of bar, then there are two inodes: one for foo called i1, and the other for bar called i2, and one dentry to represent the hierarchy of these two directories where i2 is the current inode and i1 is the parent inode.

A meta partition can only store the inodes and dentries of the files from the same volume. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting (see Resource Manager (Master)).

Replication

The replication during file write is performed in terms of meta partitions. The replication consistency is ensured by a revision of the Raft consensus protocol called the MultiRaft, which has the advantage of reduced heartbeat network traffic comparing to the original version.

Failure Recovery

The in-memory meta partitions are persisted to the local disk by snapshots and logs for backup and recovery. Some techniques such as log compaction are used to reduce the log files sizes and shorten the recovery time.

It is worth noting that, a failure that happens during a metadata operation could result an orphan inode with which has no dentry to be associated. The memory and disk space occupied by this inode can be hard to free. To minimize the chance of this case to happen, the client always issues a retry after a failure until the request succeeds or the maximum retry limit is reached.

Data Subsystem

The data subsystem is optimized for the storage of large and small files, which can be accessed in a sequential or random fashion.

Data Subsystem Architecture

Features

  • Large File Storage

For large files, the contents are stored as a sequence of one or multiple extents, which can be distributed across different data partitions on different data nodes. Writing a new file to the extent store always causes the data to be written at the zero-offset of a new extent, which eliminates the need for the offset within the extent. The last extent of a file does not need to fill up its size limit by padding (i.e., the extent does not have holes), and never stores the data from other files.

  • Small File Storage

The contents of multiple small files are aggregated and stored in a single extent, and the physical offset of each file content in the extent is recorded in the corresponding meta node. ChubaoFS relies on the punch hole interface, textit{fallocate()}footnote{url{http://man7.org/linux/man-pages/man2/fallocate.2.html}}, to textit{asynchronous} free the disk space occupied by the to-be-deleted file. The advantage of this design is to eliminate the need of implementing a garbage collection mechanism and therefore avoid to employ a mapping from logical offset to physical offset in an extent~cite{haystack}. Note that this is different from deleting large files, where the extents of the file can be removed directly from the disk.

  • Replication

    The replication is performed in terms of partitions during file writes. Depending on the file write pattern, ChubaoFS adopts different replication strategies.

    When a file is sequentially written into ChubaoFS, a primary-backup replication protocol is used to ensure the strong consistency with optimized IO throughput.

    _images/workflow-sequential-write.png

    When overwriting an existing file portion during random writes, we employ a MultiRaft-based replication protocol, which is similar to the one used in the metadata subsystem, to ensure the strong consistency.

    _images/workflow-overwriting.png

  • Failure Recovery

    Because of the existence of two different replication protocols, when a failure on a replica is discovered, we first start the recovery process in the primary-backup-based replication by checking the length of each extent and making all extents aligned. Once this processed is finished, we then start the recovery process in our MultiRaft-based replication.

HTTP APIs

API Method Parameters Description
/disks GET N/A Get disk list and informations.
/partitions GET N/A Get parttion list and infomartions.
/partition GET partitionId[int] Get detail of specified partition.
/extent GET partitionId[int]&extentId[int] Get extent informations.
/stats GET N/A Get status of the datanode.

Object Subsystem (ObjectNode)

The object subsystem provides S3-compatible object storage interface. It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.

Structure

_images/cfs-object-subsystem-structure.png

The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.

ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster which it belong to directly without any volume-mount operation.

Features

  • Provides object storage interface compatible with native Amazon S3 SDKs.
  • Fusion storage expose two type interface (POSIX and S3-compatible).
  • Stateless and high scalability

Semantic Transform

Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.

POSIX Object Storage
Cluster Region
Volume Bucket
Path Key

Example:

_images/cfs-object-subsystem-semantic.png

Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.

User

Before using the object storage service, you need to create a user through the Master. While creating users, AccessKey and SecretKey will be generated for each user.

ChubaoFS uses the field Owner of the volume as the user ID. There are two ways to create users:

  • When creating a volume through the API of the Master, if there is no user with the same name as the owner of the volume in the cluster, a user with the user ID of Owner will be automatically created.
  • Create a user by calling the user management API of the Master.

Authentication

The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with user creation, which can be obtained through the Master API. The AccessKey is a 16-character string unique in the entire ChubaoFS cluster.

The user has all access permissions to the volume owned by him. Users can grant other users specified permissions to access volumes under their own names. The permissions are divided into the following three categories:

  • Readonly or readwrite permission.
  • Permission for a single operation, such as GetObject, PutObject, etc.
  • Custom permission.

When a user uses the object storage service to execute a certain operation, ChubaoFS will identify whether the user has the corresponding permission.

Invisible Temporary Data

In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.

Object Mode Conflict (Important)

The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.

But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.

The object key like this is conflict in ChubaoFS.

When the object being operated has conflicts with the existing object due to the above mode inconsistency, the ObjectNode will return a 409 status code to the client.

Supported S3 Features

  • File object operations.
  • Directory object operations.
  • Multipart upload.
  • Parallel download for high-level SDK APIs.
  • Tagging for bucket and object.
  • User-defined metadata for object.
  • IP address and network segment black and white list for bucket ACL.
  • Signature Algorithm V2 and V4.
  • Cross-Origin Resource Sharing (CORS).

Unsupported S3 Features

  • Version
  • Restore deleted objects
  • Locking objects
  • Lifecycle configuration for bucket and object.
  • Hosting Websites
  • Encryption
  • BitTorrent

Supported APIs

API Reference
AbortMultipartUpload https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
CompleteMultipartUpload https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
CopyObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
CreateBucket https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html
CreateMultipartUpload https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html
DeleteBucket https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html
DeleteBucketCors https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketCors.html
DeleteBucketPolicy https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketPolicy.html
DeleteBucketTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketTagging.html
DeleteObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html
DeleteObjects https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
DeleteObjectTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html
GetBucketAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html
GetBucketCors https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketCors.html
GetBucketLocation https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html
GetBucketPolicy https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html
GetBucketTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketTagging.html
GetObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
GetObjectAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html
GetObjectTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html
HeadBucket https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
HeadObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html
ListBuckets https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html
ListMultipartUploads https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html
ListObjects https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html
ListObjectsV2 https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
ListParts https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html
PutBucketAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAcl.html
PutBucketCors https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html
PutBucketPolicy https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html
PutBucketTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketTagging.html
PutObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
PutObjectAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html
PutObjectTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html
UploadPart https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html

Supported SDKs

Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

Name Language Link
AWS SDK for Java Java https://aws.amazon.com/sdk-for-java/
AWS SDK for JavaScript JavaScript https://aws.amazon.com/sdk-for-browser/
AWS SDK for JavaScript in Node.js JavaScript https://aws.amazon.com/sdk-for-node-js/
AWS SDK for Go Go https://docs.aws.amazon.com/sdk-for-go/
AWS SDK for PHP PHP https://aws.amazon.com/sdk-for-php/
AWS SDK for Ruby Ruby https://aws.amazon.com/sdk-for-ruby/
AWS SDK for .NET .NET https://aws.amazon.com/sdk-for-net/
AWS SDK for C++ C++ https://aws.amazon.com/sdk-for-cpp/
Boto3 Python http://boto.cloudhackers.com

Client

The client runs on each container executing application code and exposes a file system interface to applications and can access a mounted volume via a user-space file system interface such as FUSE.

Client Caching

The client process runs entirely in the user space with its own cache, which has been used in the following cases.

To reduce the communication with the resource manager, the client caches the addresses of the available meta and data partitions assigned to the mounted volume, which can be obtained at the startup, and periodically synchronizes this available partitions with the resource manager.

To reduce the communication with the meta nodes, the client also caches the returned inodes and dentries when creating new files, as well as the data partition id, the extent id and the offset, after the file has been written to the data node successfully. When a file is opened for read/write, the client will force the cache metadata to be synchronous with the meta node.

To reduce the communication with the data nodes, the client caches the most recently identified leader. Our observation is that, when reading a file, the client may not know which data node is the current leader because the leader could change after a failure recovery. As a result, the client may try to send the read request to each replica one by one until a leader is identified. However, since the leader does not change frequently, by caching the last identified leader, the client can have minimized number of retries in most cases.

Integration with FUSE

The ChubaoFS client has been integrated with FUSE to provide a file system interface in the user space. In the past, low performance is considered the main disadvantage of such user-space file systems. But over the years, FUSE has made several improvement on its performance such as multithreading and write-back cache. In the future, we plan to develop our own POSIX-compliant file system interface in the kernel space to completely eliminate the overhead from FUSE.

Currently the write-back cache feature does not work well in ChubaoFS due to the following reason. The default write behavior of FUSE is called directIO, which bypasses the kernel’s page cache. This results in performance problems on writing small files as each write pushes the file data to the user daemon. The solution FUSE implemented was to make the page cache support a write-back policy that aggregates small data first, and then make writes asynchronous. With that change, file data can be pushed to the user daemon in larger blobs at a time. However, in real production, we found that the write-back cache is not very useful, because a write operation usually invoke another process that tries to balance the dirty pages (pages in the main memory that have been modified during writing data to disk are marked as “dirty” and have to be flushed to disk before they can be freed), which incurs extra overhead. This overhead becomes more obvious when small files are continuously written through FUSE.

Resource Manager (Master)

The cluster contains dataNodes,metaNodes,vols,dataPartitions and metaPartitions,they are managed by master server. The master server caches the metadata in mem,persist to GoLevelDB,and ensure consistence by raft protocol.

The master server manages dataPartition id to dataNode server mapping,metaPartition id to metaNode server mapping.

At lease 3 master nodes are required in respect to high availability.

Features

  • Multi-tenant, Resource Isolation
  • dataNodes,metaNodes shared,vol owns dataPartition and metaPartition exclusive
  • Asynchronous communication with dataNode and metaNode

Configurations

ChubaoFS use JSON as configuration file format.

Properties
Key Type Description Mandatory
role string Role of process and must be set to master Yes
ip string host ip Yes
listen string Http port which api service listen on Yes
prof string golang pprof port Yes
id string identy different master node Yes
peers string the member information of raft group Yes
logDir string Path for log file storage Yes
logLevel string Level operation for logging. Default is error. No
retainLogs string the number of raft logs will be retain. Yes
walDir string Path for raft log file storage. Yes
storeDir string Path for RocksDB file storage,path must be exist Yes
clusterName string The cluster identifier Yes
exporterPort int The prometheus exporter port No
consulAddr string The consul register addr for prometheus exporter No
metaNodeReservedMem string If the metanode memory is below this value, it will be marked as read-only. Unit: byte. 1073741824 by default. No

Example:

{
 "role": "master",
 "id":"1",
 "ip": "10.196.59.198",
 "listen": "17010",
 "prof":"17020",
 "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",
 "retainLogs":"20000",
 "logDir": "/cfs/master/log",
 "logLevel":"info",
 "walDir":"/cfs/master/data/wal",
 "storeDir":"/cfs/master/data/store",
 "exporterPort": 9500,
 "consulAddr": "http://consul.prometheus-cfs.local",
 "clusterName":"chubaofs01",
 "metaNodeReservedMem": "1073741824"
}

Start Service

nohup ./master -c config.json > nohup.out &

Meta Subsystem

Metanode is the manager of meta partitions and replicated by MultiRaft. Each metanode manages various of partitions. Each partition covers an inode range, and maintains two in-memory btrees: inode btree and dentry btree.

At lease 3 meta nodes are required in respect to high availability.

Properties
Key Type Description Mandatory  
role string Role of process and must be set to metanode Yes  
listen string Listen and accept port of the server Yes  
prof string Pprof port Yes  
localIP string IP of network to be choose No. If not specified, the ip address used to communicate with the master is used.  
logLevel string Level operation for logging. Default is error No  
metadataDir string MetaNode store snapshot directory Yes  
logDir string Log directory Yes  
raftDir string Raft wal directory Yes  
raftHeartbeatPort string Raft heartbeat port Yes  
raftReplicaPort string Raft replicate port Yes  
consulAddr string Addresses of monitor system No  
exporterPort string Port for monitor system No  
masterAddr string Addresses of master server Yes  
totalMem string Max memory metadata used. The value needs to be higher than the value of metaNodeReservedMem in the master configuration. Unit: byte Yes  

Example:

{
     "role": "metanode",
     "listen": "17210",
     "prof": "17220",
     "logLevel": "debug",
     "metadataDir": "/cfs/metanode/data/meta",
     "logDir": "/cfs/metanode/log",
     "raftDir": "/cfs/metanode/data/raft",
     "raftHeartbeatPort": "17230",
     "raftReplicaPort": "17240",
     "consulAddr": "http://consul.prometheus-cfs.local",
     "exporterPort": 9501,
     "totalMem":  "8589934592",
     "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"
     ]
 }

Notice

  • listen, raftHeartbeatPort, raftReplicaPort can’t be modified after boot startup first time;
  • Above config would be stored under directory raftDir in constcfg file. If need modified forcely，you must delete this file manually;
  • These configuration items associated with master’s metanode infomation . If they have been modified, master would’t be found old metanode;

Data Subsystem

How To Start DataNode

Start a DataNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file. At least 4 data nodes are required in respect to high availability.

nohup cfs-server -c datanode.json &

Configurations

Properties
Key Type Description Mandatory
role string Role of process and must be set to datanode Yes
listen string Port of TCP network to be listen Yes
localIP string IP of network to be choose No,If not specified, the ip address used to communicate with the master is used.
prof string Port of HTTP based prof and api service Yes
logDir string Path for log file storage Yes
logLevel string Level operation for logging. Default is error No
raftHeartbeat string Port of raft heartbeat TCP network to be listen Yes
raftReplica string Port of raft replicate TCP network to be listen Yes
raftDir string Path for raft log file storage No
consulAddr string Addresses of monitor system No
exporterPort string Port for monitor system No
masterAddr string slice Addresses of master server Yes
disks string slice
Format: PATH:RETAIN.
PATH: Disk mount point. RETAIN: Retain space. (Ranges: 20G-50G.)
 Yes

Example:

{
    "role": "datanode",
    "listen": "17310",
    "prof": "17320",
    "logDir": "/cfs/datanode/log",
    "logLevel": "info",
    "raftHeartbeat": "17330",
    "raftReplica": "17340",
    "raftDir": "/cfs/datanode/log",
    "consulAddr": "http://consul.prometheus-cfs.local",
    "exporterPort": 9502,
    "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"
    ],
     "disks": [
        "/data0:10737418240",
        "/data1:10737418240"
    ]
}

Notice

  • listen, raftHeartbeat, raftReplica can’t be modified after boot startup first time.
  • Above config would be stored under directory raftDir in constcfg file. If need modified forcely, you must delete this file manually.
  • These configuration items associated with master’s datanode infomation. If they have been modified, master would’t be found old datanode.

Object Subsystem (ObjectNode)

How To start ObjectNode

Start a ObjectNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file.

nohup cfs-server -c objectnode.json &

Note: If you do not intend to use the object storage service, you do not need to start the ObjectNode.

Configurations

Object Node using JSON format configuration file.

Properties

Key Type Description Mandatory
role string Role of process and must be set to objectnode Yes
listen string Listen and accept port of the server Yes
domains string slice
Domain of S3-like interface which makes wildcard domain support
Format: DOMAIN
 No
logDir string Log directory Yes
logLevel string
Level operation for logging.
Default: error
 No
masterAddr string slice
Format: HOST:PORT.
HOST: Hostname, domain or IP address of master (resource manager).
PORT: port number which listened by this master
 Yes
authNodes string slice
Format: HOST:PORT.
HOST: Hostname, domain or IP address of AuthNode.
PORT: port number which listened by this AuthNode
 Yes
exporterPort string Port for monitor system No
prof string Pprof port Yes

Example:

{
     "role": "objectnode",
     "listen": "17410",
     "domains": [
         "object.cfs.local"
     ],
     "logDir": "/cfs/Logs/objectnode",
     "logLevel": "info",
     "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"

     ],
     "exporterPort": 9503,
     "prof": "7013"
}

Fetch Authentication Keys

First, you need create a user through User API, and then get keys information. Refer to User.

You can also use Command Line Interface (CLI) tool to create and get user’s AccessKey and SecretKey:

$ cli user create [USER_ID]
$ cli user info [USER_ID]

Using Object Storage Interface

Object Subsystem (ObjectNode) provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

For detail about list of supported APIs, see Supported S3-compatible APIs at Object Subsystem (ObjectNode)

For detail about list of supported SDKs, see Supported SDKs at Object Subsystem (ObjectNode)

Using S3cmd

Use s3cmd to access the ObjectNode deployed locally.

Installation

Install the s3cmd from https://s3tools.org/s3cmd .

Configuration

Edit s3cmd configuration file $HOME/.s3cfg

host_base = 127.0.0.1
host_bucket = 127.0.0.1
use_https = False
access_key = YOUR_ACCESS_KEY
secret_key = YOUR_SECRET_KEY

Example: making a bucket (volume)

s3cmd mb s3://my_volume
Bucket 's3://my_volume/' created

Example: uploading an local file to ChubaoFS

s3cmd put data.dat s3://my_volume
upload: 'data.dat' -> 's3://my_volume/data.dat'

Example: listing buckets (volumes)

s3cmd ls
2020-05-10 15:29 s3://my_volume

Example: listing files stored in a ChubaoFS volume

s3cmd ls s3://my_volume
                         DIR    s3://my_volume/backup/
2020-05-10  15:31   10485760    s3://my_volume/data.dat
2020005-10  15:33         10    s3://my_volume/hello.txt

Example: deleting file stored in a ChubaoFS volume

s3cmd rm s3://my_volume/data.dat
delete: 's3://my_volume/data.dat'

Detail usage for s3cmd see https://s3tools.org/usage .

Using AWS Java SDK

Use AWS Java SDK to access the ObjectNode deployed locally.

Install by Maven:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.10.71</version>
</dependency>

Example: uploading file to ChubaoFS volume (PutObject)

Regions clientRegion = Region.of("*** Region name ***"); // Setup region (the cluster name)
String endpoint = "http://127.0.0.1"; // Setup endpoint
String accessKey = "*** Access Key ***"; // Setup AccessKey
String secretKey = "*** Secret Key ***"; // Setup SecretKey
String bucketName = "*** Bucket name ***"; // Setup target bucket (ChubaoFS Volume)
String objectKey = "*** File object key name ***"; // Setup object key
[]byte data = "*** Example File Data as String **".getBytes();

try {
    // Setup credential
    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client
    S3Configuration configuration = S3Configuration.builder()
        .chunkedEncodingEnabled(true)
        .build();
    S3Client client = S3Client.builder()
        .region(region)
        .credentialsProvider(credentialsProvider)
        .endpointOverride(URI.create(endpoint))
        .serviceConfiguration(configuration)
        .build();

    // Upload file
    PutObjectRequest request = PutObjectRequest.builder()
        .bucket(bucketName)
        .key(objectKey)
        .build();
    RequestBody body = RequestBody.fromBytes(data);
    s3Client.putObject(request, body)
} catch (Exception e) {
    e.printStackTrace();
}

Example: downloading file stored in ChubaoFS volume (GetObject)

Regions clientRegion = Region.of("*** Region name ***"); // Setup region
String endpoint = "http://127.0.0.1"; // Setup endpoint
String accessKey = "*** Access Key ***"; // Setup AccessKey
String secretKey = "*** Secret Key ***"; // Setup SecretKey
String bucketName = "*** Bucket name ***"; // Setup target bucket (ChubaoFS Volume)
String objectKey = "*** File object key name ***"; // Setup object key

try {
    // Setup credential
    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client
    S3Configuration configuration = S3Configuration.builder()
        .chunkedEncodingEnabled(true)
        .build();
    S3Client client = S3Client.builder()
        .region(region)
        .credentialsProvider(credentialsProvider)
        .endpointOverride(URI.create(endpoint))
        .serviceConfiguration(configuration)
        .build();

    // Get file data
    GetObjectRequest request = GetObjectRequest.builder()
        .bucket(bucketName)
        .key(objectKey)
        .build();

    InputStream is = s3Client.getObject(request)
    while (true) {
        if (is.read() == -1) {
            break
        }
    }
} catch (Exception e) {
    e.printStackTrace();
}

Using AWS Python SDK (Boto3)

Use AWS Python SDK (Boto3) to access the ObjectNode deployed locally.

Install Boto3 by PIP:

$ pip install boto3

Example: uploading file to ChubaoFS volume (PutObject)

import boto3

endpoint = " ** endpoint url ** "  # example: http://127.0.0.1
region_name = " ** region name ** "
access_key = " ** your access key ** "  # your access key
secret_key = " ** your secret key ** "  # your secret key
bucket = " ** your bucket (volume) ** "  # your volume name
key = " ** your object key (file path in CFS) ** "  # your object name

def put_file():
    session = boto3.Session(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)
    client = session.client(service_name="s3", region_name=region_name, endpoint_url=endpoint)
    client.put_object(Bucket=bucket, Key=key, Body=bytes("hello world"))

Example: downloading file stored in ChubaoFS volume (GetObject)

import boto3

endpoint = " ** endpoint url ** "  # example: http://127.0.0.1
region_name = " ** region name ** "
access_key = " ** your access key ** "  # your access key
secret_key = " ** your secret key ** "  # your secret key
bucket = " ** your bucket (volume) ** "  # your volume name
key = " ** your object key (file path in CFS) ** "  # your object name

def get_file():
    session = boto3.Session(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)
    client = session.client(service_name="s3", region_name=region_name, endpoint_url=endpoint)
    result = client.get_object(Bucket=bucket, Key=key)
    print(result["Body"].read())

Client

Prerequisite

Insert FUSE kernel module and install libfuse.

modprobe fuse
yum install -y fuse

Prepare Config File

fuse.json

{
  "mountPoint": "/cfs/mountpoint",
  "volName": "ltptest",
  "owner": "ltptest",
  "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",
  "logDir": "/cfs/client/log",
  "logLevel": "info",
  "profPort": "27510"
}
Supported Configurations
Name Type Description Mandatory
mountPoint string Mount point Yes
volName string Volume name Yes
owner string Owner name as authentication Yes
masterAddr string Resource manager IP address Yes
logDir string Path to store log files No
logLevel string Log level：debug, info, warn, error No
profPort string Golang pprof port No
exporterPort string Performance monitor port No
consulAddr string Performance monitor server address No
lookupValid string Lookup valid duration in FUSE kernel module, unit: sec No
attrValid string Attr valid duration in FUSE kernel module, unit: sec No
icacheTimeout string Inode cache valid duration in client No
enSyncWrite string Enable DirectIO sync write, i.e. make sure data is fsynced in data node No
autoInvalData string Use AutoInvalData FUSE mount option No
rdonly bool Mount as read-only file system No
writecache bool Leverage the write cache feature of kernel FUSE. Requires the kernel FUSE module to support write cache. No
keepcache bool Keep kernel page cache. Requires the writecache option is enabled. No
token string Specify the capability of a client instance. No
readRate int Read Rate Limit. Unlimited by default. No
writeRate int Write Rate Limit. Unlimited by default. No
followerRead bool Enable read from follower. False by default. No
accessKey string Access key of user who owns the volume. No
secretKey string Secret key of user who owns the volume. No
disableDcache bool Disable Dentry Cache. False by default. No
subdir string Mount sub directory. No
fsyncOnClose bool Perform fsync upon file close. True by default. No
maxcpus int The maximum number of available CPU cores. Limit the CPU usage of the client process. No

Mount

Use the example fuse.json, and client is mounted on the directory /mnt/fuse. All operations to /mnt/fuse would be performed on the backing distributed file system.

./cfs-client -c fuse.json

Unmount

It is recommended to use standard Linux umount command to terminate the mount.

Monitor

ChubaoFS use prometheus as metrics collector. It simply config as follow in master, metanode, datanode, client’s config file：

{
    "exporterPort": 9505,
    "consulAddr": "http://consul.prometheus-cfs.local"
}
  • exporterPort：prometheus exporter Port. when set, can export prometheus metrics from URL(http://$hostip:$exporterPort/metrics). If not set, prometheus exporter will unavailable；
  • consulAddr: consul register address, it can work with prometheus to auto discover deployed ChubaoFS nodes, if not set, consul register will not work.

Using grafana as prometheus metrics web front：

_images/cfs-grafana-dashboard.png

Also, user can use prometheus alertmanager to capture and route alerts to the correct receiver. please visit prometheus alertmanger web-doc

Metrics

  • Cluster

    • The number of nodes: MasterCount , MetaNodeCount , DataNodeCount , ObjectNodeCount
    • The number of clients: ClientCount
    • The number of volumes: VolumeCount
    • The size of nodes: DataNodeSize , MetaNodeSize
    • The used ratio of nodes: DataNodeUsedRatio , MetaNodeUsedRatio
    • The number of inactive nodes: DataNodeInactive , MetaNodesInactive
    • The capacity of volumes: VolumeTotalSize
    • The used ratio of volumes: VolumeUsedRatio
    • The number of error disks: DiskError

  • Volume

    • The used size of volumes: VolumeUsedSize
    • The used ratio of volumes: VolumeUsedRatio
    • The capacity change rate of volumes: VolumeSizeRate

  • Master

    • The number of invalid masters: master_nodes_invalid
    • The number of invalid metanodes: metanode_inactive
    • The number of invalid datanodes:： datanode_inactive
    • The number of invalid clients:： fuseclient_inactive

  • MetaNode

    • The duration of each operation (Time) and the number of operations per second (Ops) on the metanode, which can be selected from the MetaNodeOp drop-down list.

  • DataNode

    • The duration of each operation (Time) and the number of operations per second (Ops) on the datanode, which can be selected from the DataNodeOp drop-down list.

  • ObjectNode

    • The duration of each operation (Time) and the number of operations per second (Ops) on the objectnode, which can be selected from the objectNodeOp drop-down list.

  • FuseClient

    • The duration of each operation (Time) and the number of operations per second (Ops) on the client, which can be selected from the fuseOp drop-down list.

Recommended focus metrics: cluster status, node or disk failure, total size, growth rate, etc.

Grafana DashBoard Config

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 21,
  "iteration": 1578879879321,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 40,
      "panels": [],
      "title": "Cluster",
      "type": "row"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "description": "master node total count of cluster",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 0,
        "y": 1
      },
      "id": 38,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"master\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "MasterCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 4,
        "y": 1
      },
      "id": 42,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"metanode\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "MetanodeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 8,
        "y": 1
      },
      "id": 41,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=\"$cluster\", app=\"$app\", role=\"dataNode\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "DatanodeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 12,
        "y": 1
      },
      "id": 86,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"fuseclient\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "ClientCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "description": "Cluster volume total count",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 15,
        "y": 1
      },
      "id": 93,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "VolumeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "decimals": null,
      "format": "dateTimeFromNow",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 18,
        "y": 1
      },
      "id": 113,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "process_start_time_seconds{cluster=\"$cluster\", app=\"$app\", instance=~\"$instance\"}*1000",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "ProcessStartTime",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 5
      },
      "id": 115,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(go_info{app=\"$app\", cluster=\"$cluster\"}) by (cluster, app, role)",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{cluster}}-{{role}}",
          "refId": "A"
        },
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"$role\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "[[role]]-count",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "NodeAlive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 5
      },
      "id": 179,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "sum"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0)",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "D"
        },
        {
          "expr": "cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{addr}}_{{path}}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DiskError",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 5
      },
      "id": 180,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "B"
        },
        {
          "expr": "sum(cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetaNodesInactive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 11
      },
      "id": 181,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum(cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeInactive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 11
      },
      "id": 177,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/used_ratio.*/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "TotalGB",
          "refId": "B"
        },
        {
          "expr": "cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "UsedGB",
          "refId": "A"
        },
        {
          "expr": "cfs_master_dataNodes_increased_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "IncreasedGB",
          "refId": "C"
        },
        {
          "expr": "sum(cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}) / sum(cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "UsedRatio",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "percentunit",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 11
      },
      "id": 92,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_master_metaNodes_total_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "total_GB",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_used_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "used_GB",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_increased_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "increased_GB",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 17
      },
      "id": 175,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_ratio.*/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_vol_usage_ratio{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_usage_ratio",
          "refId": "B"
        },
        {
          "expr": "cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_total_GB",
          "refId": "A"
        },
        {
          "expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_used_GB",
          "refId": "C"
        },
        {
          "expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"} / cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"} ",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_used_ratio",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "VolumeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "percentunit",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 17
      },
      "id": 90,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "vol_count",
          "refId": "M"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "VolumeCount",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 17
      },
      "id": 73,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Open",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Lookup",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaNodeHeartbeat{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "NodeHeartbeat",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReadDir",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReleaseOpen",
          "refId": "E"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaSetattr{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Setattr",
          "refId": "F"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Truncate",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Open_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Lookup_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReadDir_QPS",
          "refId": "K"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Truncate_QPS",
          "refId": "L"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReleaseOpen_QPS",
          "refId": "J"
        },
        {
          "expr": "sum(rate([[app]]cfs_metanode_OpMetaSetattr{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Setattr_QPS",
          "refId": "M"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 23
      },
      "id": 71,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaBatchInodeGet{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "BatchGet",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Create",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Delete",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Evict",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "GetInode",
          "refId": "E"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LinkInode",
          "refId": "F"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateInode_QPS",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteInode_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "EvictInode_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "GetInode_QPS",
          "refId": "J"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LinkInode_QPS",
          "refId": "K"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpBatchInodeGet{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "BantchInodeGet_QPS",
          "refId": "L"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "TruncateInode",
          "refId": "M"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "TruncateInode_QPS",
          "refId": "N"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UnlinkInode",
          "refId": "O"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UnlinkInnode_QPS",
          "refId": "P"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeInodeOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 23
      },
      "id": 45,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDentryCreateDentry",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDentry",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UpdateDentry",
          "refId": "C"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDentry_QPS",
          "refId": "D"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDentry_QPS",
          "refId": "E"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UpdateDentry_QPS",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeDentryOperation",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 23
      },
      "id": 79,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_dataNode_OpCreateExtent{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateExtent",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "MarkDeleteExtent",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "RandomWrite",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AppendWrite",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Read",
          "refId": "F"
        },
        {
          "expr": "sum(rate([[app]]_datanode_OpCreateExtent{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateExtent_QPS",
          "refId": "E"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "MarkDeleteExtent_QPS",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "RandomWrite_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AppendWrite_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Read_QPS",
          "refId": "J"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeExtentOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 29
      },
      "id": 70,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpCreateMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Create",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpLoadMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Load",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetapartitionOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 29
      },
      "id": 75,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_dataNode_OpLoadDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LoadDataPartition",
          "refId": "G"
        },
        {
          "expr": "sum([[app]]_dataNode_OpDataNodeHeartbeat{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DataNodeHeartbeat",
          "refId": "F"
        },
        {
          "expr": "sum([[app]]_dataNode_OpCreateDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDataPartition",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_dataNode_OpAddDataPartitionRaftMember{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AddDataPartitionRaftMember",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_dataNode_OpDeleteDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDataPartition",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatapartitionOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 35
      },
      "id": 34,
      "panels": [],
      "title": "Master",
      "type": "row"
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 0,
        "y": 36
      },
      "id": 162,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "master_count",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"master\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "[[cluster]]_master_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "count(go_info{cluster=~\"$cluster\", role=~\"master\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "[[cluster]]_master_count",
          "refId": "G"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[role]]_nodes",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 7,
        "y": 36
      },
      "id": 163,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"metanode\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "meta_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_metanode_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 0,
        "y": 43
      },
      "id": 164,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"dataNode\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_datanode_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 7,
        "y": 43
      },
      "id": 165,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"fuseclient\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "fuseclient_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_fuseclient_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 50
      },
      "id": 36,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 3
          },
          "id": 117,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaCreateInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeCreate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 3
          },
          "id": 44,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeBatchGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 3
          },
          "id": 119,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaLinkInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_ops_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeLink",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 11
          },
          "id": 120,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeEvict",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "description": "",
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 11
          },
          "id": 121,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpCreateMetaPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpLoadMetaPartition_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "LoadMetaPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 11
          },
          "id": 58,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpCreateMetaPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpLoadMetaPartition_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateMetaPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 19
          },
          "id": 88,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsList",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 19
          },
          "id": 50,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsAdd",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 19
          },
          "id": 126,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaNodeHeartbeat",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 27
          },
          "id": 72,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 27
          },
          "id": 125,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaLookup",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 27
          },
          "id": 122,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 35
          },
          "id": 124,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaReadDir_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReadDir",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 35
          },
          "id": 129,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReleaseOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 35
          },
          "id": 116,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 43
          },
          "id": 127,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaTruncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 43
          },
          "id": 128,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaSetattr",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 43
          },
          "id": 123,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaUpdateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 51
          },
          "id": 46,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "Metanode",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 51
      },
      "id": 27,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "description": "",
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 52
          },
          "id": 132,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DatanodeHeartbeat",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 52
          },
          "id": 133,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpCreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 52
          },
          "id": 137,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_dataNode_OpLoadDataPartition_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "LoadDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 60
          },
          "id": 138,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "NotifyReplicasToRepair",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 60
          },
          "id": 134,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_dataNode_OpGetAllWatermarks_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetAllWatermarks",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 60
          },
          "id": 135,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_dataNode_OpGetAppliedId_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetAppliedId",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 68
          },
          "id": 142,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "rate([[app]]dataNode_OpStreamRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "StreamRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 68
          },
          "id": 143,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Write",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 68
          },
          "id": 139,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MarkDelete",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 76
          },
          "id": 144,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "rate([[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateExtent",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 76
          },
          "id": 145,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            },
            {
              "expr": "rate([[app]]_dataNode_OpExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "ExtentRepairdRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 76
          },
          "id": 74,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "BroadcastMinAppliedID",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 84
          },
          "id": 141,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "rate([[app]]_dataNode_OpReadTinyDelete_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "ReadTinyDelete",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 84
          },
          "id": 136,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetPartitionSize",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 84
          },
          "id": 146,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpTinyExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "TinyExtentRepairdRead_{{instance}}",
              "refId": "P"
            },
            {
              "expr": "rate([[app]]_dataNode_OpTinyExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "TinyExtentRepairdRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "TinyExtentRepairdRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 92
          },
          "id": 131,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DecommissionDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 92
          },
          "id": 140,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "RandomWrite",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "Datanode",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 52
      },
      "id": 66,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 5
          },
          "id": 64,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReadDir",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 5
          },
          "id": 157,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 5
          },
          "id": 156,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DeleteInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 12
          },
          "id": 160,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 12
          },
          "id": 68,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 12
          },
          "id": 102,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "rate([[app]]_fuseclient_StreamWrite_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamWrite_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamRead_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamSyncWrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "SyncWrite",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamSyncOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "SyncOverwrite",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Overwrite",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaExtentsList_count{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_count_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsList{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsList",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 19
          },
          "id": 69,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaExtentsAdd_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_count_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsList{instance=~\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetExtents",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsAdd",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 19
          },
          "id": 148,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaLookup",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 19
          },
          "id": 67,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaBatchInodeGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 26
          },
          "id": 152,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileCreate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 26
          },
          "id": 158,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 26
          },
          "id": 155,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "EvictInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 33
          },
          "id": 153,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileSync",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 33
          },
          "id": 100,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_fileread{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "fileread_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_fileread_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "fileread_count_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 33
          },
          "id": 154,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileWrite",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 40
          },
          "id": 147,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaTruncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 40
          },
          "id": 149,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr_cout{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaSetattr",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 40
          },
          "id": 150,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 47
          },
          "id": 106,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "FlushInRead_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_StreamPrepareReadRequests{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "PrepareReadRequest_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FlushInRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 47
          },
          "id": 161,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaUpdateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 47
          },
          "id": 98,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_StreamWrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Write",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_StreamRead{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Read",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_StreamOverwrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Overwrite",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_StreamCreateExtent{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "CreateExtent",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_StreamSyncWrite{instance=~\"$instance\",cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "SyncWrite",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_StreamSyncOverwrite{instance=~\"$instance\",cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "SyncOverwrite",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "StreamSDKLatency",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 54
          },
          "id": 159,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Truncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 54
          },
          "id": 151,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "FlushInRead_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_PrepareReadRequest{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "PrepareReadRequest_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "PrepareReadRequest",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "FuseClient",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 53
      },
      "id": 60,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 2
          },
          "id": 61,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "go_goroutines_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "go_threads_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "go_goroutines",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 2
          },
          "id": 169,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "go_goroutines_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "go_threads_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "go_threads",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 2
          },
          "id": 63,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_gc_duration_seconds{instance=~\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "seconds_{{quantile}}",
              "refId": "A"
            },
            {
              "expr": "rate(go_gc_duration_seconds_count{instance=~\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "gc_rate",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "gc_duration",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "s",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 8
          },
          "id": 109,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_resident_memory",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 8
          },
          "id": 166,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_virtual_memory",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 8
          },
          "id": 167,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_virtual_memory_max",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 14
          },
          "id": 110,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "open_fds_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "max_fds_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "open_fds",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 14
          },
          "id": 168,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "open_fds_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "max_fds_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "max_fds",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 14
          },
          "id": 111,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", role=\"$role\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "cpu_seconds_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_cpu",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "s",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 20
          },
          "id": 62,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "alloc_bytes",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 20
          },
          "id": 172,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "heap_alloc",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 20
          },
          "id": 171,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "heap_inuse",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 26
          },
          "id": 170,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "sys_bytes",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 26
          },
          "id": 173,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "other_sys",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 26
          },
          "id": 174,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "mcache_sys",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "GoRuntime",
      "type": "row"
    }
  ],
  "refresh": false,
  "schemaVersion": 16,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {
          "selected": true,
          "text": "cfs",
          "value": "cfs"
        },
        "hide": 2,
        "includeAll": false,
        "label": "App",
        "multi": false,
        "name": "app",
        "options": [
          {
            "selected": true,
            "text": "cfs",
            "value": "cfs"
          }
        ],
        "query": "cfs",
        "type": "custom"
      },
      {
        "allValue": null,
        "current": {
          "tags": [],
          "text": "spark",
          "value": "spark"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Cluster",
        "multi": false,
        "name": "cluster",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\"}, cluster)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "fuseclient",
          "value": "fuseclient"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Role",
        "multi": false,
        "name": "role",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\", cluster=~\"$cluster\"}, role)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "10.170.6.247:9500",
          "value": "10.170.6.247:9500"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Instance",
        "multi": false,
        "name": "instance",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\", role=~\"$role\", cluster=~\"$cluster\"}, instance)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "10.170.6.247",
          "value": "10.170.6.247"
        },
        "datasource": "default",
        "hide": 2,
        "includeAll": false,
        "label": "Host",
        "multi": false,
        "name": "hostip",
        "options": [],
        "query": "label_values(go_info{instance=~\"$instance\", cluster=~\"$cluster\"}, instance)",
        "refresh": 1,
        "regex": "/([^:]+):.*/",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "tags": [],
          "text": "clickhourse",
          "value": "clickhourse"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Vol",
        "multi": false,
        "name": "vol",
        "options": [],
        "query": "label_values(cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}, volName)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "chubaofs",
  "uid": "J8XJyOmZk",
  "version": 484
}

Tune FUSE Performance

Fetch Linux kernel source code

Download the corresponding src rpm, and use the following commands to install source code.

rpm -i kernel-3.10.0-327.28.3.el7.src.rpm 2>&1 | grep -v exist
cd ~/rpmbuild/SPECS
rpmbuild -bp --target=$(uname -m) kernel.spec

The source code will be installed in ~/rpmbuild/BUILD/

Optimize FUSE linux kernel module

In order to achieve maximum throughput performance, several FUSE kernel parameters have to be modified, such as FUSE_MAX_PAGES_PER_REQ and FUSE_DEFAULT_MAX_BACKGROUND.

Update source code according to the following lines.

/* fs/fuse/fuse_i.h */
#define FUSE_MAX_PAGES_PER_REQ 256

/* fs/fuse/inode.c */
#define FUSE_DEFAULT_MAX_BACKGROUND 32

Build against current running Linux kernel

yum install kernel-devel-3.10.0-327.28.3.el7.x86_64

cd ~/rpmbuild/BUILD/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/fuse
make -C /lib/modules/`uname -r`/build M=$PWD

Install kernel module

cp fuse.ko /lib/modules/`uname -r`/kernel/fs/fuse

rmmod fuse
depmod -a
modprobe fuse

Run Cluster on Docker

Under the docker directory, a helper tool called run_docker.sh is provided to run ChubaoFS with docker-compose.

To start a minimal ChubaoFS cluster from scratch, note that /data/disk is arbitrary, and make sure there are at least 30G available space.

$ docker/run_docker.sh -r -d /data/disk

If client starts successfully, use mount command in client docker shell to check mount status:

$ mount | grep chubaofs

Open http://127.0.0.1:3000 in browser, login with admin/123456 to view grafana monitor metrics.

Or run server and client seperately by following commands:

$ docker/run_docker.sh -b
$ docker/run_docker.sh -s -d /data/disk
$ docker/run_docker.sh -c
$ docker/run_docker.sh -m

For more usage:

$ docker/run_docker.sh -h

Prometheus and Grafana confg can be found in docker/monitor directory.

Resource Manager (Master) API

Cluster

Overview

curl -v "http://10.196.59.198:17010/admin/getCluster" | python -m json.tool

Display the base information of the cluster, such as the detail of metaNode, dataNode, vol and so on.

response

{
    "Name": "test",
    "LeaderAddr": "10.196.59.198:17010",
    "DisableAutoAlloc": false,
    "Applied": 225,
    "MaxDataPartitionID": 100,
    "MaxMetaNodeID": 3,
    "MaxMetaPartitionID": 1,
    "DataNodeStatInfo": {},
    "MetaNodeStatInfo": {},
    "VolStatInfo": {},
    "BadPartitionIDs": {},
    "BadMetaPartitionIDs": {},
    "MetaNodes": {},
    "DataNodes": {}
}

Freeze

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"

If cluster is freezed, the vol never allocates dataPartitions automatically.

Parameters
Parameter Type Description
enable bool if enable is true, the cluster is freezed

Statistics

curl -v "http://10.196.59.198:17010/cluster/stat"

Show cluster space information by zone.

response

Topology

curl -v "http://10.196.59.198:17010/topo/get"

Show cluster topology information by zone.

response

[
    {
        "Name": "zone1",
        "Status": "available",
        "NodeSet": {
            "700": {
                "DataNodeLen": 0,
                "MetaNodeLen": 0,
                "MetaNodes": [],
                "DataNodes": []
            }
        }
    },
    {
        "Name": "zone2",
        "Status": "available",
        "NodeSet": {
            "800": {
                "DataNodeLen": 0,
                "MetaNodeLen": 0,
                "MetaNodes": [],
                "DataNodes": []
            }
        }
    }
]

Update Zone

curl -v "http://10.196.59.198:17010/zone/update?name=zone1&enable=false"

Set the status of the zone to available or unavailable.

Parameters
Parameter Type Description
name string zone name
enable bool if enable is true, the cluster is available

Get Zone

curl -v "http://10.196.59.198:17010/zone/list"

Get name and status of all zones.

response

[
    {
        "Name": "zone1",
        "Status": "available",
        "NodeSet": {}
    },
    {
        "Name": "zone2",
        "Status": "available",
        "NodeSet": {}
    }
]

Volume

Create

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=100&owner=cfs&mpCount=3"
Allocate a set of data partition and a meta partition to the user.
Default create 10 data partition and 3 meta partition when create volume.
ChubaoFS uses the Owner parameter as the user ID. When creating a volume, if there is no user named the owner of the volume, a user with the user ID same as Owner will be automatically created; if a user named Owner already exists in the cluster, the volume will be owned by the user. For details, please see:: doc: /admin-api/master/user
Parameters
Parameter Type Description Mandatory Default
name string volume name Yes None
capacity int the quota of vol, unit is GB Yes None
owner string the owner of vol, and user ID of a user Yes None
mpCount int the amount of initial meta partitions No 3
enableToken bool whether to enable the token mechanism to control client permissions No false
replicaNum int the number of replicas, the value is 2 or 3 No 3
size int the size of data partitions, unit is GB No 120
followerRead bool enable read from follower No false
crossZone bool cross zone or not. If it is true, parameter zoneName must be empty No false
zoneName string specified zone No default (if crossZone is false)

Delete

curl -v "http://10.196.59.198:17010/vol/delete?name=test&authKey=md5(owner)"

Mark the vol status to MarkDelete first, then delete data partition and meta partition asynchronous, finally delete meta data from persist store.

While deleting the volume, the policy information related to the volume will be deleted from all user information.

Parameters
Parameter Type Description
name string volume name
authKey string calculates the MD5 value of the owner field as authentication information

Get

curl -v "http://10.196.59.198:17010/client/vol?name=test&authKey=md5(owner)" | python -m json.tool

Show the base information of the vol, such as name, the detail of data partitions and meta partitions and so on.

Parameters
Parameter Type Description
name string volume name
authKey string calculates the MD5 value of the owner field as authentication information

response

{
    "Name": "test",
    "Owner": "user",
    "Status": "0",
    "FollowerRead": "true",
    "MetaPartitions": {},
    "DataPartitions": {},
    "DataPartitions": {},
    "CreateTime": 0
}

Stat

curl -v http://10.196.59.198:17010/client/volStat?name=test

Show the status information of volume.

Parameters
Parameter Type Description
name string volume name

response

{
    "Name": "test",
    "TotalSize": 322122547200000000,
    "UsedSize": 155515112832780000,
    "UsedRatio": "0.48",
    "EnableToken": false
}

Update

curl -v "http://10.196.59.198:17010/vol/update?name=test&capacity=100&authKey=md5(owner)"

Increase the quota of volume, or adjust other parameters.

Parameters
Parameter Type Description Mandatory
name string volume name Yes
authKey string calculates the MD5 value of the owner field as authentication information Yes
capacity int the quota of vol, unit is GB Yes
zoneName string update zone name Yes
enableToken bool whether to enable the token mechanism to control client permissions. False by default. No
replicaNum int number of replicas, the value is 2 or 3 No
followerRead bool enable read from follower No

List

curl -v "http://10.196.59.198:17010/vol/list?keywords=test"

List all volumes information, and can be filtered by keywords.

Parameters
Parameter Type Description Mandatory
keywords string get volumes information which contains this keyword No

response

[
   {
       "Name": "test1",
       "Owner": "cfs",
       "CreateTime": 0,
       "Status": 0,
       "TotalSize": 155515112832780000,
       "UsedSize": 155515112832780000
   },
   {
       "Name": "test2",
       "Owner": "cfs",
       "CreateTime": 0,
       "Status": 0,
       "TotalSize": 155515112832780000,
       "UsedSize": 155515112832780000
   }
]

Add Token

curl -v "http://10.196.59.198:17010/token/add?name=test&tokenType=1&authKey=md5(owner)"

Add the token that controls read and write permissions.

Parameters
Parameter Type Description
name string the name of vol
tokenType int 1 is readonly token, 2 is readWrite token
authKey string calculates the MD5 value of the owner field as authentication information

Update Token

curl -v "http://10.196.59.198:17010/token/update?name=test&token=xx&tokenType=1&authKey=md5(owner)"

Update token type.

Parameters
Parameter Type Description
name string the name of vol
token string the token value
tokenType int 1 is readonly token, 2 is readWrite token
authKey string calculates the MD5 value of the owner field as authentication information

Delete Token

curl -v "http://10.196.59.198:17010/token/delete?name=test&token=xx&authKey=md5(owner)"

Delete specified token.

Parameters
Parameter Type Description
name string the name of vol
token string the token value
authKey string calculates the MD5 value of the owner field as authentication information

Get Token

curl -v "http://10.196.59.198:17010/token/get?name=test&token=xx"

Show token information.

Parameters
Parameter Type Description
name string the name of vol
token string the token value

response

{
    "TokenType":2,
    "Value":"siBtuF9hbnNqXzJfMTU48si3nzU4MzE1Njk5MDM1NQ==",
    "VolName":"test"
}

Meta Partition

Create

curl -v "http://10.196.59.198:17010/metaPartition/create?name=test&start=10000"

Split meta partition manually. If max meta partition of the vol which range is [0,end) and end larger than start parameter, old meta partition range will be [0,start], new meta partition will be [start+1,end).

Parameters
Parameter Type Description
name string the name of vol
start uint64 the start value of meta partition which will be create

Get

curl -v "http://10.196.59.198:17010/metaPartition/get?id=1" | python -m json.tool

Show base information of meta partition, such as id, start, end and so on.

Parameters
Parameter Type Description
id uint64 the id of meta partition

response

{
    "PartitionID": 1,
    "Start": 0,
    "End": 9223372036854776000,
    "MaxNodeID": 1,
    "VolName": "test",
    "Replicas": {},
    "ReplicaNum": 3,
    "Status": 2,
    "IsRecover": true,
    "Hosts": {},
    "Peers": {},
    "Zones": {},
    "MissNodes": {},
    "LoadResponse": {}
}

Decommission

curl -v "http://10.196.59.198:17010/metaPartition/decommission?id=13&addr=10.196.59.202:17210"

Remove the replica of meta partition, and create new replica asynchronous.

Parameters
Parameter Type Description
id uint64 the id of meta partition
addr string the addr of replica which will be decommission

Load

curl -v "http://10.196.59.198:17010/metaPartition/load?id=1"

Send load task to the metaNode which meta partition locate on, then check the crc of each replica in the meta partition.

Parameters
Parameter Type Description
id uint64 the id of data partition

Data Partition

Create

curl -v "http://10.196.59.198:17010/dataPartition/create?count=400&name=test"

Create a set of data partition.

Parameters
Parameter Type Description
count int the num of dataPartitions will be create
name string the name of vol

Get

curl -v "http://10.196.59.198:17010/dataPartition/get?id=100"  | python -m json.tool

Get information of the specified data partition.

Parameters
Parameter Type Description
id uint64 the id of data partition

response

{
    "PartitionID": 100,
    "LastLoadedTime": 1544082851,
    "ReplicaNum": 3,
    "Status": 2,
    "Replicas": {},
    "Hosts": {},
    "Peers": {},
    "Zones": {},
    "MissingNodes": {},
    "VolName": "test",
    "VolID": 2,
    "FileInCoreMap": {},
    "FilesWithMissingReplica": {}
}

Decommission

curl -v "http://10.196.59.198:17010/dataPartition/decommission?id=13&addr=10.196.59.201:17310"

Remove the replica of data partition, and create new replica asynchronous.

Parameters
Parameter Type Description
id uint64 the id of data partition
addr string the addr of replica which will be decommission

Load

curl -v "http://10.196.59.198:17010/dataPartition/load?id=1"

Send load task to the dataNode which data partition locate on, then check the crc of each file in the data partition asynchronous.

Parameters
Parameter Type Description
id uint64 the id of data partition

Offline Disk

curl -v "http://10.196.59.198:17010/disk/decommission?addr=10.196.59.201:17310&disk=/cfs1"

Synchronously offline all the data partitions on the disk, and create a new replica for each data partition in the cluster.

Parameters
Parameter Type Description
addr string replica address
disk string disk path

Master Management

Add

curl -v "http://10.196.59.198:17010/raftNode/add?addr=10.196.59.197:17010&id=3"

Add a new master node to master raft group.

Parameters
Parameter Type Description
addr string the addr of master server, format is ip:port
id uint64 the node id of master server

Remove

curl -v "http://10.196.59.198:17010/raftNode/remove?addr=10.196.59.197:17010&id=3"

Remove the master node from master raft group.

Parameters
Parameter Type Description
addr string the addr of master server, format is ip:port
id uint64 the node id of master server

User

Create

curl -H "Content-Type:application/json" -X POST --data '{"id":"testuser","pwd":"12345","type":3}' "http://10.196.59.198:17010/user/create"

Create a user in the cluster to access object storage service. When the cluster starts, the root user is automatically created (the value of type is 0x1).

ChubaoFS treats every Owner of volume as a user. For example, if the value of Owner is testuser when creating a volume, the volume is owned by user testuser.

If there is no user ID with the same value as the Owner when creating the volume, the user named the value of Owner will be automatically created when creating the volume.

body key
Key Type Description Range Mandatory Default
id string user ID Consists of letters, numbers and underscores, no more than 20 characters Yes None
pwd string user’s password Unlimited No ChubaoFSUser
ak string Access Key Consists of 16-bits letters and numbers No Random value
sk string Secret Key Consists of 32-bits letters and numbers No Random value
type int user type 2: [admin] / 3: [normal user] Yes None

Delete

curl -v "http://10.196.59.198:17010/user/delete?user=testuser"

Delete the specified user in the cluster.

Parameters
Parameter Type Description
user string user ID

Get

Show basic user information, including user ID, Access Key, Secret Key, list of owned volumes, list of permissions granted by other users, user type, creation time.

The field policy shows the volumes which the user has permission, of which own_vols indicates that volumes owned by the user, and authorized_vols indicates the volume authorized by other users to the user with restrictions.

There are two ways to obtain:

Query by User ID
curl -v "http://10.196.59.198:17010/user/info?user=testuser" | python -m json.tool
Parameters
Parameter Type Description
user string user ID
Query by Access Key
curl -v "http://10.196.59.198:17010/user/akInfo?ak=0123456789123456" | python -m json.tool
Parameters
Parameter Type Description
ak string Access Key

response

{
     "user_id": "testuser",
     "access_key": "0123456789123456",
     "secret_key": "ZVY5RHlrnOrCjImW9S3MajtYZyxSegcf",
     "policy": {
         "own_vols": ["vol1"],
         "authorized_vols": {
             "ltptest": [
                 "perm:builtin:ReadOnly",
                 "perm:custom:PutObjectAction"
             ]
         }
     },
     "user_type": 3,
     "create_time": "2020-05-11 09:25:04"
}

List Users

curl -v "http://10.196.59.198:17010/user/list?keywords=test" | python -m json.tool

Query information about all users in a cluster whose user ID contains the keyword.

Parameters
Parameter Type Description
keywords string check user ID contains this or not

Update

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","access_key":"KzuIVYCFqvu0b3Rd","secret_key":"iaawlCchJeeuGSnmFW72J2oDqLlSqvA5","type":3}' "http://10.196.59.198:17010/user/update"

Update the specified user’s information, including access key, secret key and user type.

body key
Key Type Description Mandatory
user_id string user ID value after updating Yes
access_key string Access Key value after updating No
secret_key string Secret Key value after updating No
type int user type value after updating No

Update Permission

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","volume":"vol","policy":["perm:builtin:ReadOnly","perm:custom:PutObjectAction"]}' "http://10.196.59.198:17010/user/updatePolicy"

Update the specified user’s permission to a volume. There are three types of values for policy:

  • Grant read-only or read-write permission, the value is perm:builtin:ReadOnly or perm:builtin:Writable.
  • Grant a permission of the specified action, the format is action:oss:XXX, take GetObject action as an example, the value of policy is action:oss:GetObject.
  • Grant a custom permission, the format is perm:custom:XXX, where XXX is customized by the user.

After the permissions are specified, the user can only access the volume within the specified permissions when using the object storage. If the user already has permissions for this volume, this operation will overwrite the original permissions.

body key
Key Type Description Mandatory
user_id string user ID to be set Yes
volume string volume name to be set Yes
policy string slice policy to be set Yes

Remove Permission

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","volume":"vol"}' "http://10.196.59.198:17010/user/removePolicy"

Remove all permissions of a specified user for a volume.

body key
Key Type Description Mandatory
user_id string user ID to be deleted Yes
volume string volume name to be deleted Yes

Transfer Volume

curl -H "Content-Type:application/json" -X POST --data '{"volume":"vol","user_src":"user1","user_dst":"user2","force":"true"}' "http://10.196.59.198:17010/user/transferVol"

Transfer the ownership of the specified volume. This operation removes the specified volume from the owner_vols of source user name and adds it to the owner_vols of target user name; At the same time, the value of the field Owner in the volume structure will also be updated to the target user ID.

body key
Key Type Description Mandatory
volume string Volume name to be transfered Yes
user_src string Original owner of the volume, and must be the same as the Owner of the volume Yes
user_dst string Target user ID after transferring Yes
force bool Force to transfer the volume. If the value is set to true, even if the value of user_src is different from the value of the owner of the volume, the volume will also be transferred to the target user No

Meta Node API

Meta Partition

Get Partitions

curl -v  http://10.196.59.202:17210/getPartitions

Get all meta-partition base information of the metanode.

Get Partition by ID

curl -v http://10.196.59.202:17210/getPartitionById?pid=100

Get the specified partition information, this result contains: leader address, raft group peer and cursor.

Parameters
Parameter Type Description
pid integer meta-partition id

Inode

Get Inode

curl -v http://10.196.59.202:17210/getInode?pid=100&ino=1024

Get inode information

Parameters
Parameter Type Description
pid integer meta-partition id
ino integer inode id

Get Extents by Inode

curl -v http://10.196.59.202:17210/getExtentsByInode?pid=100&ino=1024

Get inode all extents information

Parameters
Parameter Type Description
pid integer meta-partition id
ino integer inode id

Get All Inodes

curl -v http://10.196.59.202:17210/getAllInodes?pid=100

Get all inodes of the specified partition

Parameters
Parameter Type Description
pid integer meta-partition id

Dentry

Get Dentry

curl -v 'http://10.196.59.202:17210/getDentry?pid=100&name=""&parentIno=1024'

Get dentry information

Parameters
Parameter Type Description
pid integer meta partition id
name string file or directory name
parentIno integer file or directory parent directory inode

Get Directory

curl -v "http://10.196.59.202:17210/getDirectory?pid=100&parentIno=1024"

Get all files of the parent inode is 1024

Parameters
Parameter Type Description
pid integer partition id
ino integer inode id

Get All Dentry

curl -v "http://10.196.59.202:17210/getAllDentry?pid=100"
Parameters
Parameter Type Description
pid integer partition id

Command Line Interface

CLI

Using the Command Line Interface (CLI) can manage the cluster conveniently. With this tool, you can view the status of the cluster and each node, and manage each node, volumes, or users.

As the CLI continues to improve, it will eventually achieve 100% coverage of the APIs of each node in the cluster.

Compile and Configure

After downloading the ChubaoFS source code, execute the command go build in the directory chubaofs/cli to generate cli.

At the same time, a configuration file named .cfs-cli.json will be generated in the directory root, and the master address can be changed to the current cluster master address. You can also get or set the master address by executing the command ./cli config info or ./cli config set.

Usage

In the directory chubaofs/cli, execute the command ./cli --help or ./cli -h to get the CLI help document.

CLI is mainly divided into six types of management commands.

Commands List
Command description
cli cluster Manage cluster components
cli metanode Manage meta nodes
cli datanode Manage data nodes
cli datapartition Manage data partitions
cli volume, vol Manage cluster volumes
cli user Manage cluster users
Cluster Management
./cli cluster info          #Show cluster summary information

./cli cluster stat          #Show cluster status information
MetaNode Management
./cli metanode list         #List information of meta nodes
DataNode Management
./cli datanode list         #List information of meta nodes
DataPartition Management
./cli datapartition info [VOLUME] [Partition ID]        #Get information of data partition
Volume Management
./cli volume create [VOLUME NAME] [USER ID] [flags]     #Create a new volume
Flags:
    --capacity uint                                     #Specify volume capacity [Unit: GB] (default 10)
    --dp-size  uint                                     #Specify size of data partition size [Unit: GB] (default 120)
    --follower-read                                     #Enable read form replica follower (default true)
    --mp-count int                                      #Specify init meta partition count (default 3)
    --replicas int                                      #Specify volume replicas number (default 3)
    -y, --yes                                           #Answer yes for all questions

./cli volume delete [VOLUME NAME] [flags]               #Delete a volume from cluster
Flags:
    -y, --yes                                           #Answer yes for all questions

./cli volume info [VOLUME NAME] [flags]                 #Show volume information
Flags:
    -d, --data-partition                                #Display data partition detail information
    -m, --meta-partition                                #Display meta partition detail information

./cli volume add-dp [VOLUME] [NUMBER]                   #Create and add more data partition to a volume

./cli volume list                                       #List cluster volumes

./cli volume transfer [VOLUME NAME] [USER ID] [flags]   #Transfer volume to another user. (Change owner of volume)
Flags：
    -f, --force                                         #Force transfer without current owner check
    -y, --yes                                           #Answer yes for all questions
User Management
./cli user create [USER ID] [flags]         #Create a new user
Flags：
    --access-key string                     #Specify user access key for object storage interface authentication
    --secret-key string                     #Specify user secret key for object storage interface authentication
    --password string                       #Specify user password
    --user-type string                      #Specify user type [normal | admin] (default "normal")
    -y, --yes                               #Answer yes for all questions

./cli user delete [USER ID] [flags]         #Delete specified user
Flags：
    -y, --yes                               #Answer yes for all questions

./cli user info [USER ID]                   #Show detail information about specified user

./cli user list                             #List cluster users

./cli user perm [USER ID] [VOLUME] [PERM]   #Setup volume permission for a user
                                            #The value of [PERM] is READONLY, RO, READWRITE, RW or NONE

./cli user update [USER ID] [flags]         #Update information about specified user
Flags：
    --access-key string                     #Update user access key
    --secret-key string                     #Update user secret key
    --user-type string                      #Update user type [normal | admin]
    -y, --yes                               #Answer yes for all questions

Use Cases

ChubaoFS is a distributed file system that is compatible with most POSIX file system semantics. When ChubaoFS is mounted, it can be as simple as using a local file system. Basically, it can be used in any case where a file system is needed, replacing the local file system, and realizing infinitely expandable storage without physical boundaries. It has been applied in various scenarios, and the following are some of the extracted scenes.

Machine Learning

Disadvantages of using a local disk to store training data sets:

  • The local disk space is small, and there are multiple models. The training data set of each model reaches the TB level. If you use a local disk to store the training data set, you need to reduce the size of the training data set.
  • Training data sets need to be updated frequently, requiring more disk space.
  • Risk of loss of training data set if machine crashes.

The advantages of using ChubaoFS to store training data sets:

  • Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
  • Multiple replicas of data to ensure high data reliability without worrying about losing data.
  • Compatible with POSIX file system interface, no changes required by the application.

ElasticSearch

Using local disks to store data often encounters the following problems:

  • Disk usage is uneven and disk IO cannot be fully utilized.
  • Local disk space is limited.

The advantages of using ChubaoFS as a backend storage:

  • Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
  • Disk IO usage is uniform and disk IO is fully utilized.
  • Ensure data is highly reliable without worrying about losing data.

Nginx Log Storage

Do you often worry about running out of local disk space? With ChubaoFS, you can store datas in a distributed file system without worrying about running out of disk space. If you use a local disk to store logs, you may often worry about the following issues:

  • Docker local disk space is small.
  • If the docker container crashes, the log is lost and unrecoverable.
  • The mixed deployment of physical machine and docker machine is difficult to manage and has high operation and maintenance cost.

The advantages of using ChubaoFS to store nginx logs are:

  • The disk space is unlimited and easy to expand. The disk capacity is automatically expanded according to the percentage of disk usage. The storage system can be expanded on demand, which greatly saves storage costs.
  • Ensure data is highly reliable and do not worry about losing data.
  • Multiple replicas to solve the problem of unable to write to the log caused by disk error and datanode crashes.
  • Compatible with the POSIX file system interface, no changes required by the application.
  • The operation and maintenance of ChubaoFS are simple, and one person can easily manage the cluster of tens of thousands of machines.

Spark

In the big data set scenario, are you worried about the amount of data stored in Spark intermediate calculation results in order to carefully calculate each task? You can store the shuffle results to ChubaoFS, and no longer worry about the disk has no free space which causes the task to fail. This enables the separation of storage and computation. The pain points of using local disk to store shuffle intermediate results:

  • Insufficient disk space.
  • Too many temporary directory files to create new files.

The advantages of using ChubaoFS:

  • Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
  • Meta node manages file metadata, which can be expanded horizontally and the number of files is unlimited.

MySQL Database Backup

Disadvantages of using OSS(Object Storage Service) to back up MySQL database:

  • Need to use the OSS SDK or RESTful API to develop backup programs which increases the difficulty of operation and maintenance.
  • If backup file fails, troubleshooting is more difficult.
  • After backing up files to OSS, it is not convenient to check whether the files are successfully uploaded.
  • Backup files are processed by multiple layers of services, which affects performance.

Advantages of using ChubaoFS to back up MySQL databases:

  • Easy to use, compatible with POSIX file interface, and can be used as a local file system.
  • Complete and detailed operation logs are stored in the local file system, making it easy to troubleshoot problems.
  • Simply execute the ls command to verify that the file was successfully uploaded.
  • Supports PageCache and WriteCache, and file read and write performance is significantly improved compared to OSS.

Performance

Environment

Cluster Information

Cluster Information
Instance Nodes CPU Memory Storage Network Description
Master 3 32 32 GB 120 GB SSD 10 Gb/s  
MetaNode 10 32 32 GB 16 x 1TB SSD 10 Gb/s hybrid deployment
DataNode 10 32 32 GB 16 x 1TB SSD 10 Gb/s hybrid deployment

Volume Setup

#!/bin/bash
# create volume
curl "http://${MASTER}/admin/createVol?name=intest&owner=cfs&capacity=300000000&mpCount=10"
# increase 1500 data partitions
curl "http://${MASTER}/dataPartition/create?name=intest&count=1500"
  • Initial number of Meta Partitions: 10
  • Number of Meta Replication: 3
  • Initial number of Data Partitions: 1510
  • Number of Data Replication: 3

Small File Performance and Scalability

Small file operation performance and scalability benchmark test by mdtest.

Setup

#!/bin/bash
set -e
TARGET_PATH="/mnt/test/mdtest" # mount point of ChubaoFS volume
for FILE_SIZE in 1024 2048 4096 8192 16384 32768 65536 131072 # file size
do
mpirun --allow-run-as-root -np 512 --hostfile hfile64 mdtest -n 1000 -w $i -e $FILE_SIZE -y -u -i 3 -N 1 -F -R -d $TARGET_PATH;
done

Benchmark

Small File Benchmark
Small File Benchmark
File Size (KB) 1 2 4 8 16 32 64 128
Creation (TPS) 70383 70383 73738 74617 69479 67435 47540 27147
Read (TPS) 108600 118193 118346 122975 116374 110795 90462 62082
Removal (TPS) 87648 84651 83532 79279 85498 86523 80946 84441
Stat (TPS) 231961 263270 264207 252309 240244 244906 273576 242930

IO Performance and Scalability

IO Performance and benchmark scalability test by fio.

Note: Multiple clients mount the same volume. And the process refers to the fio process.

1. Sequential Read

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=read \  # sequential read
    -bs=128k \  # block size
    -direct=1 \ # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth

Sequential Read Bandwidth (MB/s)
Sequential Read Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 148.000 626.000 1129.000 1130.000
2 Clients 284.000 1241.000 2258.000 2260.000
4 Clients 619.000 2640.000 4517.000 4515.000
8 Clients 1193.000 4994.000 9006.000 9034.000

IOPS

Sequential Read IOPS
Sequential Read IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 1180.000 5007.000 9031.000 9040.000
2 Clients 2275.000 9924.000 18062.000 18081.000
4 Clients 4954.000 21117.000 36129.000 36112.000
8 Clients 9531.000 39954.000 72048.000 72264.000

Latency

Sequential Read Latency (Microsecond)
Sequential Read Latency (Microsecond)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 842.200 794.340 1767.310 7074.550
2 Clients 874.255 801.690 1767.370 7071.715
4 Clients 812.363 760.702 1767.710 7077.065
8 Clients 837.707 799.851 1772.620 7076.967

2. Sequential Write

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=write \ # sequential write
    -bs=128k \  # block size
    -direct=1 \ # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth

Sequential Write Bandwidth (MB/s)
Sequential Write Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 52.200 226.000 956.000 1126.000
2 Clients 104.500 473.000 1763.000 2252.000
4 Clients 225.300 1015.000 2652.000 3472.000
8 Clients 480.600 1753.000 3235.000 3608.000

IOPS

Sequential Write IOPS
Sequential Write IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 417 1805 7651 9004
2 Clients 835 3779 14103 18014
4 Clients 1801 8127 21216 27777
8 Clients 3841 14016 25890 28860

Latency

Sequential Write Latency (Microsecond)
Sequential Write Latency (Microsecond)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 2385.400 2190.210 2052.360 7081.320
2 Clients 2383.610 2081.850 2233.790 7079.450
4 Clients 2216.305 1947.688 2946.017 8842.903
8 Clients 2073.921 2256.120 4787.496 17002.425

3. Random Read

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=randread \ # random read
    -bs=4k \       # block size
    -direct=1 \    # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth

Random Read Bandwidth (MB/s)
Random Read Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 6.412 39.100 216.000 534.000
2 Clients 14.525 88.100 409.000 1002.000
4 Clients 33.242 200.200 705.000 1693.000
8 Clients 59.480 328.300 940.000 2369.000

IOPS

Random Read IOPS
Random Read IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 1641 10240 56524.800 140288
2 Clients 3718 23142.4 107212.8 263168
4 Clients 8508 52428.8 184627.2 443392
8 Clients 15222 85072.8 246681.6 621056

Latency

Random Read Latency (Microsecond)
Random Read Latency (Microsecond)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 603.580 395.420 287.510 466.320
2 Clients 532.840 351.815 303.460 497.100
4 Clients 469.025 317.140 355.105 588.847
8 Clients 524.709 382.862 530.811 841.985

4. Random Write

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=randwrite \ # random write
    -bs=4k \        # block size
    -direct=1 \     # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth

Random Write Bandwidth (MB/s)
Random Write Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 3.620 17.500 118.000 318.000
2 Clients 7.540 44.800 230.000 476.000
4 Clients 16.245 107.700 397.900 636.000
8 Clients 39.274 208.100 487.100 787.100

IOPS

Random Write IOPS
Random Write IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 926.000 4476.000 31027.200 83251.200
2 Clients 1929.000 11473.000 60313.600 124620.800
4 Clients 4156.000 27800.000 104243.200 167014.400
8 Clients 10050.000 53250.000 127692.800 206745.600

Latency

Random Write Latency
Random Write Latency
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 1073.150 887.570 523.820 784.030
2 Clients 1030.010 691.530 539.525 1042.685
4 Clients 955.972 575.183 618.445 1552.205
8 Clients 789.883 598.393 1016.185 2506.424

Metadata Performance and Scalability

Metadata performance and scalability benchmark test by mdtest.

Setup

#!/bin/bash
TEST_PATH=/mnt/cfs/mdtest # mount point of ChubaoFS volume
for CLIENTS in 1 2 4 8 # number of clients
do
mpirun --allow-run-as-root -np $CLIENTS --hostfile hfile01 mdtest -n 5000 -u -z 2 -i 3 -d $TEST_PATH;
done

Dir Creation

Dir Creation
Dir Creation Benchmark
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 448.618 2421.001 14597.97 43055.15
2 Clients 956.947 5917.576 28930.431 72388.765
4 Clients 2027.02 13213.403 54449.056 104771.356
8 Clients 4643.755 27416.904 89641.301 119542.62

Dir Removal

Dir Removal
Dir Stat Benchmark
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 399.779 2118.005 12351.635 34903.672
2 Clients 833.353 5176.812 24471.674 50242.973
4 Clients 1853.617 11462.927 46413.313 91128.059
8 Clients 4441.435 24133.617 74401.336 115013.557

Dir Stat

Dir Stat
Dir Removal Benchmark
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 283232.761 1215309.524 4231088.104 12579177.02
2 Clients 572834.143 2169669.058 8362749.217 18120970.71
4 Clients 1263474.549 3333746.786 10160929.29 31874265.88
8 Clients 2258670.069 8715752.83 22524794.98 77533648.04

File Creation

File Creation
File Creation Benchmark
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 448.888 2400.803 13638.072 27785.947
2 Clients 925.68 5664.166 25889.163 50434.484
4 Clients 2001.137 12986.968 50330.952 91387.825
8 Clients 4479.831 25933.437 86667.966 112746.199

File Removal

File Removal
File Removal Benchmark
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 605.143 3678.138 18631.342 47035.912
2 Clients 1301.151 8365.667 34005.25 64860.041
4 Clients 3032.683 14017.426 50938.926 80692.761
8 Clients 7170.386 32056.959 68761.908 88357.563

Tree Creation

Tree Creation
Tree Creation Benchmark
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 305.778 229.562 86.299 23.917
2 Clients 161.31 211.119 76.301 24.439
4 Clients 260.562 223.153 81.209 23.867
8 Clients 350.038 220.744 81.621 17.144

Tree Removal

Tree Removal
Tree Removal Benchmark
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 137.462 70.881 31.235 7.057
2 Clients 217.026 113.36 23.971 7.128
4 Clients 231.622 113.539 30.626 7.299
8 Clients 185.156 83.923 20.607 5.515

Integrity

  • Linux Test Project / fs

Workload

  • Database backup

  • Java application logs

  • Code git repo

  • Database systems

    MyRocks, MySQL Innodb, HBase,

Scalability

  • Volume Scalability: tens to millions of cfs volumes
  • Metadata Scalability: a big volume with billions of files/directories

Contributing to ChubaoFS

Bug Reports

Please make sure the bug is not already reported by searching the repository with reasonable keywords. Then, open an issue with steps to reproduce.

Patch Guidelines

In order for the patch to be accepted with higher possibility, there are a few things you might want to pay attention to:

runltp -f fs -d [MOUNTPOINT]
  • A good commit message describing the bug fix or the new feature is preferred.
  • DCO is required, so please add Signed-off-by to the commit.

Credits

Sections of this documents have been borrowed from CoreDNS and Fluentd

FAQ

1. How does ChubaoFS compare with its alternatives ?

Ceph

Ceph (https://ceph.com/) is a widely-used free-software storage platform. It can be configured for object storage, block storage, as well as file system. But many nice features provided by Ceph (e.g., various storage backends) also make it very complicated and difficult to learn and deploy. In addition, in certain scenarios such as metadata operations and small file operations, its performance in a multi-client environment can be hard to optimize.

GFS and HDFS

GFS and its open source implementation HDFS (https://github.com/apache/hadoop) are designed for storing large files with sequential access. Both of them adopt the master-slave architecture, where the single master stores all the file metadata. Unlike GFS and HDFS, ChubaoFS employs a separate metadata subsystem to provide a scalable solution for the metadata storage so that the resource manager has less chance to become the bottleneck.

Hadoop Ozone

Hadoop Ozone is a scalable distributed object storage system designed for Hadoop. It was originally proposed to solve the problem of HDFS namenode expansion. It reconstructs the namenode metadata management part of hdfs and reuses the datanode of hdfs. ChubaoFS has many of the same design concepts like ozone such as: supporting for volume isolation, compatible with both raft/master-slave synchronous replication mechanisms, implenting for s3-compatible interfaces. In addition, ChubaoFS’s POSIX fuse-client interface supports random file reading and writing, and optimizes reading and writing of small files.

Haystack

Haystack from Facebook takes after log-structured filesystems to serve long tail of requests seen by sharing photos in a large social network. The key insight is to avoid disk operations when accessing metadata. ChubaoFS adopts similar ideas by putting the file metadata into the main memory.

However, different from Haystack, the actually physical offsets instead of logical indices of the file contents are stored in the memory, and deleting a file is achieved by the punch hole interface provided by the underlying file system instead of relying on the garbage collector to perform merging and compacting regularly for more efficient disk utilization. In addition, Haystack does not guarantee the strong consistency among the replicas when deleting the files, and it needs to perform merging and compacting regularly for more efficient disk utilization, which could be a performance killer. ChubaoFS takes a different design principle to separate the storage of file metadata and contents. In this way, we can have more flexible and cost-effective deployments of meta and data nodes.

Public Cloud

Windows Azure Storage (https://azure.microsoft.com/en-us/) is a cloud storage system that provides strong consistency and multi-tenancy to the clients. Different from ChubaoFS, it builds an extra partition layer to handle random writes before streaming data into the lower level. AWS EFS (https://aws.amazon.com/efs/) is a cloud storage service that provides scalable and elastic file storage. Depending on the use cases, there could be a considerable amount of cost associated with using these cloud storage services.

Others

GlusterFS (https://www.gluster.org/) is a scalable distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. MooseFS (https://moosefs.com/) is a fault- tolerant, highly available, POSIX-compliant, and scalable distributed file system. However, similar to HDFS, it employs a single master to manage the file metadata.

2. A problem similar to undefined reference to ‘ZSTD_versionNumber’.

There are two solutions.

  • It can be compiled by adding the specified library to CGO_LDFLAGS.

For example: CGO_LDFLAGS="-L/usr/local/lib -lrocksdb -lzstd". This requires that the zstd library is also installed on other deployment machines.

  • Remove the script that automatically detects whether the ‘zstd’ library is installed.

Example of file location: rockdb-5.9.2/build_tools/build_detect_platform

The deleted content is as follows.

# Test whether zstd library is installed
    $CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
    #include <zstd.h>
    int main() {}
EOF
    if [ "$?" = 0 ]; then
        COMMON_FLAGS="$COMMON_FLAGS -DZSTD"
        PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lzstd"
        JAVA_LDFLAGS="$JAVA_LDFLAGS -lzstd"
    fi

3. Compile ChubaoFS on one machine, but it cannot be started when deployed to other machines.

First please make sure to use the PORTABLE=1 make static_lib command to compile RocksDB, then use the ldd command to check whether the dependent libraries are installed on the machine. After installing the missing libraries, execute the ldconfig command.

4. Why is the status of MetaNode “ReadOnly”?

  • Reason 1: The MetaNode heartbeat has not been received for a long time.
  • Reason 2: The used ratio of MetaNode has reached the set threshold.
  • Reason 3: The remaining available memory of MetaNode (TotalWeight minus UsedWeight) is not greater than the value of Master’s configuration metaNodeReservedMem. You can appropriately increase the MetaNode’s TotalWeight configuration parameter.
  • Reason 4: The number of MetaNode meta partitions reaches the default maximum number of partitions (10000).

5. How to change the parameters specified during volume creation?

Use Mater API updateVol to change the parameters capacity, replicaNum, zoneName, enableToken and followerRead.

6. How to launch a new disk?

Add the path of the new disk to the configuration parameter disks of DataNode, and restart the node.

Operations Guides

  1. Log information：metaPartition(xx) changeLeader to (xx)：

    Change leader. It is a normal behavior.

  2. Log information：inode count is not equal, vol[xxx], mpID[xx]：

    The number of inodes is inconsistent. When writing data, as long as two of the three replicas are successful, it will be considered successful. So there will be inconsistencies in the three replicas. Check the logs for specific reasons.

  3. Log information：clusterID[xxx] addr[xxx]_op[xx] has no response util time out：

    During the [op] operation, the connection to the node at this address timed out. Solution: Check the network connectivity; see if the service process is still there.

  4. Log information：checkFileCrcTaskErr clusterID[xxx] partitionID:xxx File:xxx badCrc On xxx:

    The file crc check failed. Check the DataNode logs for specific reasons.

  5. Can the master change IP?

    Yes. But in addition to adjusting the configuration of the Master, you also need to adjust the configuration of all functional nodes (MetaNode, DataNode, ObjectNode) and Client in the entire cluster, so that these components can communicate with the Master after the adjustment of the IP.

  6. One of the three masters is broken. Can the remaining two restart to provide services normally?

    Yes. Since the Master uses the RAFT algorithm, services can be provided normally when the number of remaining nodes exceeds 50% of the total number of nodes.

  7. How to deal with physical machine failure of DataNode?

    Refer to the DataNode API Datanode Related to decommission the specified DataNode. The offline operation will remove the specified DataNode from the resource pool of the cluster, and ChubaoFS will automatically use the available resources in the cluster to complete the data replicas loss caused by the offline DataNode.

  8. How to deal with physical machine failure of MetaNode?

    Refer to the MetaNode API Metanode Related to decommission the specified MetaNode. The offline operation will remove the specified MetaNode from the resource pool of the cluster, and ChubaoFS will automatically use the available resources in the cluster to complete the meta replicas loss caused by the offline MetaNode.