Tracking GCP RAG Data Stores

[leekahung]

If there were to be an update to our documents (i.e. new amendments to existing statutes or to add additional ordinances or laws related to housing in Oregon or specific cities), we should probably just create a new one based on how it's documented in the wiki and swap the older data store with the newer one.

We'll probably want additional documentation of that process.

[yangm2]

What do you mean? Remove them from the repository? Or remove them from the Google Cloud Storage? Or delete the deprecated data store?

Once we have a data store in production, we should avoid ever changing (including deleting) that datastore ever again, even after it is deprecated. There could be some far-in-the-future-date when we need to clean up old data stores, but that decision should be driven by resource constraints (e.g. $$$) or privacy/security concerns.

Once we deprecate a data store, it is possible that we may need to put that back into production in order to restore functionality, revert behavior, reproduce a bug, etc.

[leekahung]

Thinking more of a swap so we can revert to older data stores if needed.

[leekahung]

Renaming issue to "Swapping data stores" and updating the issue description

[yangm2]

related to #287

[yangm2]

#316

[yangm2]

I'm pinning this issue. Let's use this to document/track when we create/promote a Data Store.

[yangm2]

Data Store ID	Content	Env Var	Status	Comment
city-state-law-data-2025-edition_1771660760568	Oregon, Portland & Eugene laws	VERTEX_AI_DATASTORE (or VERTEX_AI_DATASTORE_LAWS in staging)	prd	no chunking
oregon-law-help-search-2026_1775704132316	Oregon Law Help	VERTEX_AI_DATASTORE_OREGON_LAW_HELP	staging
city-state-law-data-2025-edition-v2_1771727870732	Oregon, Portland & Eugene laws	VERTEX_AI_DATASTORE (or VERTEX_AI_DATASTORE_LAWS in staging)	prd	chunking w/ 500 token limit