这篇用供应链项目里已经跑通的表,把最常用的几段 SQL 摊开。引擎写法接近 DuckDB / Spark SQL,逻辑可迁到 Hive。
演示大屏:oss.zl4629.xyz
建模背景:分层与维度建模
010. 表从哪来
| 表 | 用途 |
|---|---|
dwd_fact_dependency | 依赖边明细 |
dwd_dim_package_version | 版本维 |
dwd_fact_vuln_affects | 漏洞影响范围 |
dwd_fact_download_daily | npm 日下载 |
dwd_fact_vuln | 漏洞主数据 |
下面 SQL 里的 ${dt} 表示业务日分区。
021. 窗口函数:挑代表版本
包日指标不能把多个采样版本简单相加。
WITH ranked AS (
SELECT
ecosystem_code,
package_name,
version,
published_at,
is_default,
row_number() OVER (
PARTITION BY ecosystem_code, package_name
ORDER BY
CASE WHEN coalesce(is_default, false) THEN 0 ELSE 1 END,
published_at DESC NULLS LAST,
version DESC
) AS rn
FROM dwd_dim_package_version
WHERE dt = '${dt}'
)
SELECT *
FROM ranked
WHERE rn = 1;这里用到了:
PARTITION BY:每个包单独排序ORDER BY多键:默认版本优先,再比时间row_number():只留一行代表版本
032. 分组聚合:直接依赖数(M01)
SELECT
d.from_ecosystem_code AS ecosystem_code,
d.from_package_name AS package_name,
d.from_version AS version,
count(*) AS direct_dep_cnt
FROM dwd_fact_dependency d
WHERE d.dt = '${dt}'
AND d.relation = 'DIRECT'
GROUP BY 1, 2, 3;再和代表版本表连接,得到包日 M01:
SELECT
r.ecosystem_code,
r.package_name,
coalesce(x.direct_dep_cnt, 0) AS direct_dep_cnt
FROM map_package_rep_version r
LEFT JOIN (
SELECT from_ecosystem_code, from_package_name, from_version, count(*) AS direct_dep_cnt
FROM dwd_fact_dependency
WHERE dt = '${dt}' AND relation = 'DIRECT'
GROUP BY 1, 2, 3
) x
ON x.from_ecosystem_code = r.ecosystem_code
AND x.from_package_name = r.package_name
AND x.from_version = r.rep_version;过滤条件写在 WHERE 里,比事后口头解释“这个数不含 INDIRECT”更稳。
043. 聚合:高危范围榜(M03)
SELECT
ecosystem_code,
package_name,
count(DISTINCT osv_id) AS high_vuln_affect_cnt
FROM dwd_fact_vuln_affects
WHERE dt = '${dt}'
AND upper(coalesce(severity_level, 'UNKNOWN')) IN ('CRITICAL', 'HIGH')
AND ecosystem_code IS NOT NULL
GROUP BY 1, 2
ORDER BY high_vuln_affect_cnt DESC
LIMIT 50;注意:这是 affects 范围级 计数。
在版本精确命中做好之前,不能把它表述成“当前版本已确认中招”。
054. 窗口滑动:近 7 日下载(M05)
SELECT
package_name,
cast(day AS date) AS day,
downloads,
sum(downloads) OVER (
PARTITION BY package_name
ORDER BY cast(day AS date)
ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
) AS download_cnt_7d
FROM dwd_fact_download_daily
WHERE ecosystem_code = 'npm';这里的关键点:
ROWS BETWEEN定义窗口范围。- 缺日是否补 0,要在口径里写死;我当前选择 不补 0,只对已有日求和。
065. 关联:种子闭包被依赖热点(M08)
SELECT
to_ecosystem_code AS ecosystem_code,
to_package_name AS package_name,
count(*) AS depended_cnt_seed
FROM dwd_fact_dependency
WHERE dt = '${dt}'
AND relation = 'DIRECT'
GROUP BY 1, 2
ORDER BY depended_cnt_seed DESC
LIMIT 50;deps.dev dependents 接口不可用时,这就是可落地的替代。
指标名必须带 seed / seed_closure,避免被理解成全网统计。
076. 数据质量:对账比“感觉对”重要
·6.1 入仓条数对账
SELECT
'dependency' AS table_name,
count(*) AS row_cnt
FROM dwd_fact_dependency
WHERE dt = '${dt}'
UNION ALL
SELECT 'vuln_affects', count(*)
FROM dwd_fact_vuln_affects
WHERE dt = '${dt}'
UNION ALL
SELECT 'download_daily', count(*)
FROM dwd_fact_download_daily
WHERE dt = '${dt}';每日写入 dq_recon,异常时先看条数跳变,再查上游。
·6.2 映射率
SELECT
count(*) FILTER (WHERE ecosystem_code IS NOT NULL) * 1.0
/ nullif(count(*), 0) AS osv_map_ratio
FROM dwd_fact_vuln_affects
WHERE dt = '${dt}';我把门禁设在 ≥ 0.99。
映射失败的行不静默丢掉,而是留在明细里并进质量报告。
·6.3 唯一性
SELECT
count(*) AS rows,
count(DISTINCT osv_id) AS uniq_ids
FROM dwd_fact_vuln
WHERE dt = '${dt}';rows != uniq_ids 时直接阻断发布。
087. 一段业务宽表 SQL
把代表版本、高危、下载拼成包日宽表,供看板取数:
SELECT
p.ecosystem_code,
p.package_name,
p.rep_version,
coalesce(d.direct_dep_cnt, 0) AS direct_dep_cnt,
coalesce(h.high_vuln_affect_cnt, 0) AS high_vuln_affect_cnt,
dl.download_cnt_7d
FROM map_package_rep_version p
LEFT JOIN /* M01 */ ... d
LEFT JOIN /* M03 */ ... h
LEFT JOIN /* M05 */ ... dl;看板不直接扫 ODS 原文,只读 DWS/ADS。
098. 我写 SQL 时的习惯
- 先写粒度注释,再写 SELECT。
- 过滤条件进 WHERE,不靠看板口头补充。
- 口径变更就升指标版本,不静默改聚合。
- 质量结果单独落表,方便复盘。
复杂不在于关键字堆叠,而在于:算对的是同一件业务事实。