数据质量中心OpenAPI

此手册用于记录大数据开发与管理平台中，DQC产品所有对外开放的OpenAPI，阅读此手册，你将了解DQC产品开放的OpenAPI能力与调用方法。

一、公共参数

1.1 响应格式

1.2 异常响应码

注意：按照统一规范，实际通过openapi网关返回的与业务异常响应码为：18+响应码 其中18为固定表示easydqc子产品，其它非业务异常码可以参考：OpenAPI通用规范说明

1.3 枚举对象

1.3.1 DatasourceTypeEnum

数据源类型

1.3.2 ScheduleTypeEnum

调度类型

1.3.3 JobStatusEnum

执行实例状态值

1.3.4 RuleLevelEnum

规则类型

1.3.5 CheckTypeEnum

校验类型

1.3.6 ComputeRuleTypeEnum

计算规则类型

1.3.7 CompareTypeEnum

校验方式

1.3.8 ReferenceValueTypeEnum

参考值类型

1.3.9 ExploreTaskRuleTypeEnum

形态探查任务规则类型枚举

1.3.10 JobInstanceEnum

1.4 基础对象

1.4.1 PageResult

分页查询结果

1.4.2 User

用户信息

1.4.3 JobRunParameter

任务运行参数，支持spark运行参数

1.5 执行实例

1.5.1 PartitionField

分区配置

1.5.2 Job

执行实例数据对象

1.5.3 JobResult

执行实例运行结果数据对象

1.5.4 TemplateRuleResult

模板规则结果对象

1.5.5 CustomRuleResult

自定义规则结果对象JobResult

1.5.6 RuleExpect

规则期望数据对象

1.5.7 RuleColumnExpect

规则字段期望数据对象

1.5.8 RuleActualValue

规则实际运行结果数据对象

1.5.9 RuleColumnActualValue

规则字段的实际运行结果数据对象

1.5.10 OpenapiRunDqTaskInstanceIdVO

规则字段的实际运行结果数据对象

1.6 质量报告

1.6.1 RankedTable

表排序对象

1.6.2 TablePg

表排序对象

1.7 形态探查

1.7.1 ExploreTask

形态探查任务列表信息

1.7.2 ExploreTaskInstance

形态探查任务实例列表信息

1.7.3 ExploreTaskTableResult

形态探查任务表级结果

1.7.4 ExploreTaskFieldResult

形态探查任务字段级结果

1.7.5 ExploreTaskEnumIntanceResult

形态探查任务字枚举实例结果

1.7.6 ExploreTaskEnumResult

形态探查任务字段级结果

1.7.7 ExploreTaskInstanceID

形态探查任务实例ID

1.7.8 ExploreTaskInstanceResult

形态探查任务实例结果详情

1.7.9 JobExecuteParam

形态探查任务实例执行参数

二、OpenAPI列表

2.1 OpenAPI总览

2.2 基础信息

2.2 执行实例

2.2.1 查询执行实例列表

POST /job/v1/list

产品版本：v2.1.0、v2.9.0

描述：查询执行实例列表，按监控开始时间、数据源类型、监控负责人、表负责人、监控名称、表名、实例类型、实例ID 查询

URL参数/请求体

Content-Type: application/json

请求示例：（注意鉴权信息需要额外添加，具体可以参考OpenAPI通用规范说明)

{

    "product": "WKEhVpR6Kb",

    "pageNum": 1,

    "clusterId": "56868",

    "pageSize": 10,

    "user": "bqOLc1RTBo",

    "datasourceType": "MYSQL",

    "taskOwners": \[

            "3hGdAMl1hP",

            "A2RdY0n3Ie",

            "eTzzcRSTXD"

    \],

    "tableOwners": \[

            "eSpXoCpxAW",

            "aOU2iCqmYz",

            "0qEm5tjEGJ"

    \],

    "scheduleType": 31641,

    "startTime": 1650542443020,

    "endTime": 1650541578929,

    "searchType": 2,

    "searchValue": "wwqCYsIcsK",

    "taskName": "T1eKScwmJX"

}

响应体

响应示例：

{

  "reqId": "sdfds8434sfdf",

  "code": 0,

  "msg": "success",

  "cost": 236,

  "result": {

    "pageSize": 100,

    "pageNum": 2,

    "totalCount": 1000,

    "totalPage": 10,

    "list": \[

      {

        "jobId": 99939,

        "taskId": 100752,

        "taskName": "tnPJMdBJ6E",

        "scheduleType": 9154,

        "datasourceType": "ORACLE",

        "datasourceId": 101194,

        "db": "Lg7RF0NyMX",

        "table": "BHwszDZpOE",

        "partitionFields": \[

          {

            "fieldType": "String",

            "fieldName": "date",

            "fieldValue": "2022-04-21"

          }

        \],

        "taskOwner": {

          "user": "ELFbM4nnHz",

          "fullName": "5q5bl9KG72"

        },

        "tableOwner": {

          "user": "ek3do9UKsG",

          "fullName": "zA64A5uHr7"

        },

        "startTime": "2022-04-22 10:00:00",

        "endTime": "2022-04-22 10:00:00",

        "status": 94392,

        "planRuleNum": 89147,

        "errorRuleNum": 58357

      }

    \]

  }

}

2.2.2 查询执行实例运行结果

GET /job/v1/result

产品版本：v2.1.0、v2.7.0、v2.9.0

描述：查询执行实例运行结果

URL参数/请求体

请求示例：（注意鉴权信息需要额外添加，具体可以参考OpenAPI通用规范说明)

?jobId=1&product=intern&clusterId=dev4&user=

响应体

响应示例：

{

  "reqId": "sdfds8434sfdf",

  "code": 0,

  "msg": "success",

  "cost": 236,

  "result": {

    "templateRules": \[

      {

        "id": 70974,

        "description": "3wX7phTccc",

        "disable": false,

        "interrupt": false,

        "columnName": "RV96kUIBbo",

        "ruleLevel": 0,

        "checkType": "-1",

        "ruleName": "88sgyMV30I",

        "actuals": \[

          {

            "sampleValue": "2orYakJpWL",

            "referenceResultValue": "tlArhi6os5",

            "computeResultValue": "Kj4UQ3wTgi",

            "compareType": "3",

            "leftValue": "1.0",

            "rightValue": "hfZwv5BKHt",

            "computeRuleType": "REFERENCE\_DIFFERENCE",

            "referenceType": "NTH\_DAYS\_LATEST",

            "referenceValue": 1

          }

        \],

        "expects": \[

          {

            "compareType": "6",

            "leftValue": "1.0",

            "rightValue": "IVOcT2rNwa",

            "computeRuleType": "REFERENCE\_DIFFERENCE",

            "referenceType": "NTH\_DAYS\_AVG",

            "referenceValue": 1

          }

        \],

        "pass": true,

        "abnormalDataSql": "select count(1) as num from \`test\`.\`test\`"

      }

    \],

    "customRules": \[

      {

        "id": 70974,

        "description": "3wX7phTccc",

        "disable": false,

        "interrupt": false,

        "columnName": "RV96kUIBbo",

        "ruleLevel": 0,

        "checkType": "-1",

        "ruleName": "88sgyMV30I",

        "actuals": \[

          {

            "sampleValue": "2orYakJpWL",

            "referenceResultValue": "tlArhi6os5",

            "computeResultValue": "Kj4UQ3wTgi",

            "compareType": "3",

            "leftValue": "1.0",

            "rightValue": "hfZwv5BKHt",

            "computeRuleType": "REFERENCE\_DIFFERENCE",

            "referenceType": "NTH\_DAYS\_LATEST",

            "referenceValue": 1

          }

        \],

        "expects": \[

          {

            "compareType": "6",

            "leftValue": "1.0",

            "rightValue": "IVOcT2rNwa",

            "computeRuleType": "REFERENCE\_DIFFERENCE",

            "referenceType": "NTH\_DAYS\_AVG",

            "referenceValue": 1

          }

        \],

        "pass": true,

        "abnormalDataSql": "select count(1) as num from \`test\`.\`test\`"

      }

    \],

    "standardRules": \[

      {

        "id": 70974,

        "description": "3wX7phTccc",

        "disable": false,

        "interrupt": false,

        "columnName": "RV96kUIBbo",

        "ruleLevel": 0,

        "checkType": "-1",

        "ruleName": "88sgyMV30I",

        "actuals": \[

          {

            "sampleValue": "2orYakJpWL",

            "referenceResultValue": "tlArhi6os5",

            "computeResultValue": "Kj4UQ3wTgi",

            "compareType": "3",

            "leftValue": "1.0",

            "rightValue": "hfZwv5BKHt",

            "computeRuleType": "REFERENCE\_DIFFERENCE",

            "referenceType": "NTH\_DAYS\_LATEST",

            "referenceValue": 1

          }

        \],

        "expects": \[

          {

            "compareType": "6",

            "leftValue": "1.0",

            "rightValue": "IVOcT2rNwa",

            "computeRuleType": "REFERENCE\_DIFFERENCE",

            "referenceType": "NTH\_DAYS\_AVG",

            "referenceValue": 1

          }

        \],

        "pass": true,

        "abnormalDataSql": "select count(1) as num from \`test\`.\`test\`"

      }

    \]

  }

}

2.2.3 运行质量监控任务

POST /job/v1/run

产品版本：v3.3.0

描述：运行质量监控任务

URL参数/请求体

请求示例：（注意鉴权信息需要额外添加，具体可以参考OpenAPI通用规范说明)

{

"product": "intern",

"clusterId": "dev4",

"user": "zhulianxin@corp.netease.com",

"taskId": "50",

"queueName": "intern\_new",

  "parameters": \[\],

  "ruleConcurrent": true,

  "ruleMerge": true,

  "noticeConditions": \[1, 2\],

  "noticeWays": \[1\]

}

响应体

响应示例：

{

  "reqId": "sdfds8434sfdf",

  "code": 0,

  "msg": "success",

  "cost": 236,

  "result": {

    "id": 111

  }

}

2.3 质量报告

2.3.1 查询今日表质量分排行

GET /quality/v1/table/rank

产品版本：v2.1.0

描述：查询今日表质量分排行

URL参数/请求体

请求示例：（注意鉴权信息需要额外添加，具体可以参考OpenAPI通用规范说明)

?datasourceType=&product=intern&clusterId=dev4&pageNum=1&pageSize=25

响应体

响应示例：

{

  "reqId": "sdfds8434sfdf",

  "code": 0,

  "msg": "success",

  "cost": 236,

  "result": {

    "pageSize": 100,

    "pageNum": 2,

    "totalCount": 1000,

    "totalPage": 10,

    "list": \[

      {

        "rank": 1,

        "datasourceType": "HIVE",

        "datasourceId": "71284",

        "db": "jyu9Zf3xUF",

        "table": "k92CYClMKR",

        "owner": {

          "fullName": "9PQ2ZFrDNo",

          "user": "4hqyKH@hzATV.rrh"

        },

        "qualityScore": "9VwxyzAK1t"

      },

      {

        "rank": 1,

        "datasourceType": "ORACLE",

        "datasourceId": "71649",

        "db": "fgIOqwbZCe",

        "table": "BS5jp7VvlF",

        "owner": {

          "fullName": "ICm4HGItV5",

          "user": "jYlTLUhl@y3gLv.dct"

        },

        "qualityScore": "tuZQg5vU1w"

      },

      {

        "rank": 1,

        "datasourceType": "ORACLE",

        "datasourceId": "72142",

        "db": "pJh3V8RvsV",

        "table": "97eAZj3QH0",

        "owner": {

          "fullName": "oaeTmD2Yah",

          "user": "VYuJxv4@WlXQR.cwr"

        },

        "qualityScore": "HQNVhOTNNA"

      }

    \]

  }

}

2.3.2 查询项目组下有表级质量报告的表列表

POST /quality/v1/table/list

产品版本：v3.8.0

描述：查询项目组下有表级质量报告的表列表

URL参数/请求体

请求示例：

{

    "pgId": 65,

    "email": "yan",

    "datasourceType": "hive",

    "tableLayerList": \["dim"\],

    "subjectFullIdList":\["608/787"\],

    "tableOwner":"zhaomin3@corp.netease.com",

    "pageNum": 1,

    "pageSize": 10

}

响应体

响应示例：

{

"code":0,

"reqId":"074fb9a4a4a74725928da65850fc4c61",

"cost":107,

"result":{

    "count":2,

    "pageNum":1,

    "pageSize":10,

    "list":\[

    {

    "catalogName":"dev4-catalog",

    "datasourceType":"hive",

    "dbName":"dev",

    "tableName":"ads\_test",

    },

    {

    "catalogName":"dev4-catalog",

    "datasourceType":"hive",

    "dbName":"jinyx\_hivedb",

    "tableName":"dqc\_demo\_07",

    }

    \],

    "hasMore":false

    }

}

2.4 形态探查

2.4.1 查询形态探查任务列表

POST /explore-task/v1/list

产品版本：v2.8.0

描述：查询形态探查任务列表，可按修改时间、任务负责人、表名、任务名称查询

URL参数/请求体

Content-Type: application/json

请求示例：

{

"product": "intern",

"clusterId": "easyops-cluster",

"pageNum": 1,

"pageSize": 10,

"user": "zhulianxin@corp.netease.com",

"modifyStartTime": 1670241780699,

"modifyEndTime": 1670241830681,

"table": "rQoj5mDFxz",

"taskOwners": \[

"DlNdSBWpV8",

"FEwNp4SUeF",

"JxjKPGWRBg"

\],

"taskName": "7zIZhfH5n0"

}

响应体

响应示例：

{

    "code": 0,

    "msg": null,

    "result": {

        "pageSize": 10,

        "pageNum": 1,

        "totalCount": 38,

        "list": \[

            {

                "taskId": "52",

                "taskName": "20221212202354\_探查hive.cwztest3.b\_cwztest2\_sub2\_ccz\_df\_a\_copy",

                "db": "cwztest3",

                "table": "cwz",

                "partitionFields": \[

                    {

                        "fieldType": "string",

                        "fieldName": "e",

                        "fieldValue": "${azkaban.flow.1.days.ago}"

                    }

                \],

                "otherFilter": "1=1 and 2=2",

                "owner": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "creator": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "modifier": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "华昊泽"

                },

                "mode": "online"

            },

            {

                "taskId": "50",

                "taskName": "20221123194639\_探查hive.cwz11\_dev.gaolong\_test",

                "db": "cwz11\_dev",

                "table": "gaolong\_test",

                "description": "sadsadasddsadasd",

                "owner": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "creator": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "modifier": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "mode": "dev"

            },

            {

                "taskId": "49",

                "taskName": "20221123193724\_探查hive.cwz11\_dev.gaolong\_test",

                "db": "cwz11\_dev",

                "table": "gaolong\_test",

                "description": "asdasd是的撒的sss",

                "owner": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "creator": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "modifier": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "mode": "online"

            },

            {

                "taskId": "48",

                "taskName": "20221116163329\_探查hive.dev45.ods\_eto\_az\_exec\_job\_1d",

                "db": "chensuan",

                "table": "test",

                "otherFilter": "1=2",

                "description": "dasdasdasdasd",

                "owner": {

                    "user": "huahaoze@corp.netease.com",

                    "fullName": "华昊泽"

                },

                "creator": {

                    "user": "huahaoze@corp.netease.com",

                    "fullName": "华昊泽"

                },

                "modifier": {

                    "user": "huahaoze@corp.netease.com",

                    "fullName": "祝联新"

                },

                "mode": "online"

            },

            {

                "taskId": "47",

                "taskName": "���",

                "db": "cwz",

                "table": "ods\_ei\_index2",

                "partitionFields": \[

                    {

                        "fieldType": "string",

                        "fieldName": "pt",

                        "fieldValue": "${azkaban.flow.1.days.ago}"

                    }

                \],

                "description": "描述",

                "owner": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "creator": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "modifier": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "mode": "dev"

            },

            {

                "taskId": "46",

                "taskName": "20221031150733\_探查hive.data\_transform.docs\_1000",

                "db": "data\_transform",

                "table": "docs\_1000",

                "owner": {

                    "user": "xuchuqiao@corp.netease.com",

                    "fullName": "徐楚翘"

                },

                "creator": {

                    "user": "xuchuqiao@corp.netease.com",

                    "fullName": "徐楚翘"

                },

                "modifier": {

                    "user": "xuchuqiao@corp.netease.com",

                    "fullName": "徐楚翘"

                },

                "mode": "dev"

            },

            {

                "taskId": "45",

                "taskName": "20221031105914\_探查hive.default.test\_a",

                "db": "default",

                "table": "test\_a",

                "owner": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "creator": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "modifier": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "mode": "dev"

            },

            {

                "taskId": "44",

                "taskName": "20221028150002\_探查hive.cwz.cwztestdmla",

                "db": "hzluzhonghao",

                "table": "sparktest",

                "description": "sss",

                "owner": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "creator": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "modifier": {

                    "user": "zengxiaomin@corp.netease.com",

                    "fullName": "曾晓敏"

                },

                "mode": "dev"

            },

            {

                "taskId": "43",

                "taskName": "20221028104257\_探查hive.intern\_new.src\_t2\_v602",

                "db": "intern\_new",

                "table": "src\_t2\_v602",

                "owner": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "creator": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "modifier": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "华昊泽"

                },

                "mode": "dev"

            },

            {

                "taskId": "42",

                "taskName": "20221028103124\_探查hive.intern\_new.src\_t1\_v602",

                "db": "intern\_new",

                "table": "src\_t1\_v602",

                "owner": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "creator": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "modifier": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "mode": "dev"

            }

        \]

    },

    "reqId": "b45bd7089131404baf4660acd0d263f3",

    "cost": 15

}

2.4.2 运行形态探查任务

POST /explore-task/v1/run

产品版本：v2.8.0

描述：运行形态探查任务，用户需要有探查任务中表的读权限和队列的使用权限

URL参数/请求体

Content-Type: application/json

请求示例：

{

"product": "intern",

"clusterId": "dev4",

"user": "zhulianxin@corp.netease.com",

"taskId": "50",

"queueName": "intern\_new",

"yarnId": "default",

"yarnName": "默认Yarn集群",

"partitionFields": \[{

"fieldName": "1",

"fieldValue": "2",

"fieldType": "string"

}\],

        "authType": "TEST"

}

响应体

响应示例

{

    "code": 0,

    "msg": null,

    "result": {

        "id": "120"

    },

    "reqId": "79d1c59bb2bf4b3881a15b386727db55",

    "cost": 444

}

2.4.3 查询形态探查任务实例列表

POST /explore-task/v1/instance/list

产品版本：v2.8.0

描述：查询形态探查任务实例列表

URL参数/请求体

Content-Type: application/json

请求示例：

{

"product": "intern",

"clusterId": "dev4",

"user": "zhulianxin@corp.netease.com",

"taskId": "43",

"pageNum": 1,

"pageSize": 3,

        "authType": "TEST"

}

响应体

响应示例

{

    "code": 0,

    "msg": null,

    "result": {

        "pageSize": 3,

        "pageNum": 1,

        "totalCount": 1,

        "list": \[

            {

                "id": "77",

                "taskId": "43",

                "taskName": "20221028104257\_探查hive.intern\_new.src\_t2\_v602",

                "queueName": "intern\_new",

                "db": "intern\_new",

                "table": "src\_t2\_v602",

                "taskOwner": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "creator": {

                    "user": "zhulianxin@corp.netease.com",

                    "fullName": "祝联新"

                },

                "startTime": "2022-10-31 10:59:43",

                "endTime": "2022-10-31 11:01:57",

                "status": 3,

                "logUrl": "https://easydata-dev.bdms.netease.com/dataquality/instance/log/77?product=intern&clusterId=dev4&execId=3581020&type=1"

            }

        \]

    },

    "reqId": "35f4d4185cd6457baa09351e9d125297",

    "cost": 20

}

2.4.4 查询形态探查任务实例结果

GET /explore-task/v1/instance-result/get

产品版本：v2.8.0

描述：查询形态探查任务实例结果

URL参数

请求示例：

product=intern&clusterId=dev4&user=zhulianxin@corp.netease.com&id=105

响应体

响应示例

{

    "code": 0,

    "msg": null,

    "result": {

        "id": "105",

        "exploreTask": {

            "taskId": "46",

            "taskName": "20221031150733\_探查hive.data\_transform.docs\_1000",

            "db": "data\_transform",

            "table": "docs\_1000",

            "createdTime": "2022-10-31 15:08:24",

            "updatedTime": "2022-11-10 16:39:23"

        },

        "jobExecuteParam": {

            "executorsNum": 4,

            "driverMemory": 2048,

            "executorMemory": 2048,

            "queueName": "intern\_1"

        },

        "creator": {

            "user": "xuchuqiao@corp.netease.com",

            "fullName": "徐楚翘"

        },

        "startTime": "2022-12-08 10:27:30",

        "endTime": "2022-12-08 10:27:31",

        "status": 5,

        "logUrl": "https://easydata-dev.bdms.netease.com/dataquality/instance/log/105?product=intern&clusterId=dev4&execId=3620016&type=1"

    },

    "reqId": "290cb2d9b49542ae8c8c1acdebbd1bb0",

    "cost": 6

}