这篇用供应链项目里已经跑通的表,把最常用的几段 SQL 摊开。引擎写法接近 DuckDB / Spark SQL,逻辑可迁到 Hive。

演示大屏:oss.zl4629.xyz
建模背景:分层与维度建模

010. 表从哪来

表用途
dwd_fact_dependency依赖边明细
dwd_dim_package_version版本维
dwd_fact_vuln_affects漏洞影响范围
dwd_fact_download_dailynpm 日下载
dwd_fact_vuln漏洞主数据

下面 SQL 里的 ${dt} 表示业务日分区。

021. 窗口函数:挑代表版本

包日指标不能把多个采样版本简单相加。

sql
WITH ranked AS (
  SELECT
    ecosystem_code,
    package_name,
    version,
    published_at,
    is_default,
    row_number() OVER (
      PARTITION BY ecosystem_code, package_name
      ORDER BY
        CASE WHEN coalesce(is_default, false) THEN 0 ELSE 1 END,
        published_at DESC NULLS LAST,
        version DESC
    ) AS rn
  FROM dwd_dim_package_version
  WHERE dt = '${dt}'
)
SELECT *
FROM ranked
WHERE rn = 1;

这里用到了:

  • PARTITION BY:每个包单独排序
  • ORDER BY 多键:默认版本优先,再比时间
  • row_number():只留一行代表版本

032. 分组聚合:直接依赖数(M01)

sql
SELECT
  d.from_ecosystem_code AS ecosystem_code,
  d.from_package_name   AS package_name,
  d.from_version        AS version,
  count(*)              AS direct_dep_cnt
FROM dwd_fact_dependency d
WHERE d.dt = '${dt}'
  AND d.relation = 'DIRECT'
GROUP BY 1, 2, 3;

再和代表版本表连接,得到包日 M01:

sql
SELECT
  r.ecosystem_code,
  r.package_name,
  coalesce(x.direct_dep_cnt, 0) AS direct_dep_cnt
FROM map_package_rep_version r
LEFT JOIN (
  SELECT from_ecosystem_code, from_package_name, from_version, count(*) AS direct_dep_cnt
  FROM dwd_fact_dependency
  WHERE dt = '${dt}' AND relation = 'DIRECT'
  GROUP BY 1, 2, 3
) x
  ON x.from_ecosystem_code = r.ecosystem_code
 AND x.from_package_name = r.package_name
 AND x.from_version = r.rep_version;

过滤条件写在 WHERE 里,比事后口头解释“这个数不含 INDIRECT”更稳。

043. 聚合:高危范围榜(M03)

sql
SELECT
  ecosystem_code,
  package_name,
  count(DISTINCT osv_id) AS high_vuln_affect_cnt
FROM dwd_fact_vuln_affects
WHERE dt = '${dt}'
  AND upper(coalesce(severity_level, 'UNKNOWN')) IN ('CRITICAL', 'HIGH')
  AND ecosystem_code IS NOT NULL
GROUP BY 1, 2
ORDER BY high_vuln_affect_cnt DESC
LIMIT 50;

注意:这是 affects 范围级 计数。
在版本精确命中做好之前,不能把它表述成“当前版本已确认中招”。

054. 窗口滑动:近 7 日下载(M05)

sql
SELECT
  package_name,
  cast(day AS date) AS day,
  downloads,
  sum(downloads) OVER (
    PARTITION BY package_name
    ORDER BY cast(day AS date)
    ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
  ) AS download_cnt_7d
FROM dwd_fact_download_daily
WHERE ecosystem_code = 'npm';

这里的关键点:

  1. ROWS BETWEEN 定义窗口范围。
  2. 缺日是否补 0,要在口径里写死;我当前选择 不补 0,只对已有日求和。

065. 关联:种子闭包被依赖热点(M08)

sql
SELECT
  to_ecosystem_code AS ecosystem_code,
  to_package_name   AS package_name,
  count(*)          AS depended_cnt_seed
FROM dwd_fact_dependency
WHERE dt = '${dt}'
  AND relation = 'DIRECT'
GROUP BY 1, 2
ORDER BY depended_cnt_seed DESC
LIMIT 50;

deps.dev dependents 接口不可用时,这就是可落地的替代。
指标名必须带 seed / seed_closure,避免被理解成全网统计。

076. 数据质量:对账比“感觉对”重要

·6.1 入仓条数对账

sql
SELECT
  'dependency' AS table_name,
  count(*)     AS row_cnt
FROM dwd_fact_dependency
WHERE dt = '${dt}'
UNION ALL
SELECT 'vuln_affects', count(*)
FROM dwd_fact_vuln_affects
WHERE dt = '${dt}'
UNION ALL
SELECT 'download_daily', count(*)
FROM dwd_fact_download_daily
WHERE dt = '${dt}';

每日写入 dq_recon,异常时先看条数跳变,再查上游。

·6.2 映射率

sql
SELECT
  count(*) FILTER (WHERE ecosystem_code IS NOT NULL) * 1.0
  / nullif(count(*), 0) AS osv_map_ratio
FROM dwd_fact_vuln_affects
WHERE dt = '${dt}';

我把门禁设在 ≥ 0.99。
映射失败的行不静默丢掉,而是留在明细里并进质量报告。

·6.3 唯一性

sql
SELECT
  count(*) AS rows,
  count(DISTINCT osv_id) AS uniq_ids
FROM dwd_fact_vuln
WHERE dt = '${dt}';

rows != uniq_ids 时直接阻断发布。

087. 一段业务宽表 SQL

把代表版本、高危、下载拼成包日宽表,供看板取数:

sql
SELECT
  p.ecosystem_code,
  p.package_name,
  p.rep_version,
  coalesce(d.direct_dep_cnt, 0) AS direct_dep_cnt,
  coalesce(h.high_vuln_affect_cnt, 0) AS high_vuln_affect_cnt,
  dl.download_cnt_7d
FROM map_package_rep_version p
LEFT JOIN /* M01 */ ... d
LEFT JOIN /* M03 */ ... h
LEFT JOIN /* M05 */ ... dl;

看板不直接扫 ODS 原文,只读 DWS/ADS。

098. 我写 SQL 时的习惯

  1. 先写粒度注释,再写 SELECT。
  2. 过滤条件进 WHERE,不靠看板口头补充。
  3. 口径变更就升指标版本,不静默改聚合。
  4. 质量结果单独落表,方便复盘。

复杂不在于关键字堆叠,而在于:算对的是同一件业务事实。