Optimize daily_package_downloads with partitioning, clustering, and bootstrap migration

[Copilot]

The daily_package_downloads table lacks partitioning and clustering, causing 153GB scans for 14-day test queries. This adds ~$20/week in unnecessary costs and will worsen as the table grows.

Changes

New optimized model

• Created daily_package_downloads_optimised.sql with:
  • PARTITION BY download_date for temporal pruning
  • CLUSTER BY package, package_version for package-specific queries
  • Bootstrap logic to copy existing 18 months of data instead of reprocessing from source (~95% cost reduction on initial deployment)

{% if is_incremental() %}
  -- Standard incremental: new data only
  SELECT ... FROM {{ ref('file_downloads') }}
  WHERE download_date >= '{{ latest_partition_date }}'
{% else %}
  -- Bootstrap: copy existing + new data
  SELECT * FROM {{ ref('daily_package_downloads') }}
  UNION ALL
  SELECT ... FROM {{ ref('file_downloads') }}
  WHERE download_date > '{{ old_table_latest_date }}'
{% endif %}

Test optimization

• Scoped tests to 14-day window via where config (153GB → 2.5GB)
• Updated references in downloads_and_vulnerabilities.sql and test files

Documentation

• MIGRATION_STRATEGY.md: Bootstrap rationale, deployment steps, rollback path
• PIPELINE_REFACTORING_ANALYSIS.md: Recommends separate daily pipeline in same repo for fresher data at similar cost

Expected impact

• Query costs: ~98% reduction for date-filtered queries
• Migration: Avoid reprocessing 18 months of PyPI source data
• Performance: Sub-second partition-pruned queries vs full table scans
• Future-ready: Enables daily refresh pipeline without expensive full builds

Original table remains unchanged for safe rollback.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Refactor the daily_package_downloads table for more efficient date and package based queries</issue_title>
<issue_description>As a maintainer
I want to minimise the query time and cost for queries involving daily_package_downloads, like merging in new data and running tests
So that I can avoid unnecessary spend and compute consumption

As a user
I want to minimise the query time and cost for queries involving daily_package_downloads
So that I can avoid unnecessary spend and compute consumption

Context

The daily_package_downloads table (defined by [models/internal/pypi/daily_package_downloads.sql])(models/internal/pypi/daily_package_downloads.sql) is currently unpartitioned and unclustered. As the table grows, queries and tests become more expensive.

The table can be considered append-only and is currently updated weekly, so the current tests can be changed to check the last week or two. At the moment that will not reduce query costs, as there is no partitioning or clustering.

Typical consumer queries are also often time-bound and/or limited to specific package(s) potentially with versions. Initial experiments partitioning by download_date and clustering by package and package_version are effective at cutting and stabilising query costs (example test over the last 14 days currently scans 153GB in this table, but only 2.5GB when executed against a copy of the table that has been partitioned and clustered like this:

CREATE TABLE sandbox_paul_brabban_us_internal.daily_package_downloads_optimised
  PARTITION BY download_date
  CLUSTER BY package, package_version
AS
SELECT
  *
FROM `pypi-vulns.published_us_internal.daily_package_downloads`;

No permissions are granted to run queries directly.

</issue_description>

<agent_instructions>Verify the following proposed implementation plan and suggest any improvements or risks that might impact success:

1. create a new dbt model daily_package_downloads_optimised replicating the current model and tests but with partitioning and clustering as described. Execute this change as a first, isolated step and verify that the tests pass. How can I best do that as a single, idempotent change in dbt?
2. update existing references in the merge script to update the new daily_package_downloads_optimised model. Verify correct functioning by running it.
3. update other references through the dbt project to point to the new optimised model. Verify correct functioning.
4. where appropriate, update test logic to check the last two weeks' data, not the whole history. Verify successful functioning and reduction in processing and costs.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Refactor the daily_package_downloads table for more efficient date and package based queries #21

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.