Add support of Google BigQuery as Data Storage

[vladflaks]

Starting discussions for requirement gathering. (or Starting a discussion for requirement gathering.)
At this moment, I am going to use MERGE for BigQuery. Trade off: this service is unavailable outside Apps Script.

[NKrivosheiev]

Hello, @vladflaks!

What do you think about using DELETE / APPEND method instead of MERGE?

It is very handy when we use it for partitioned tables in BigQuery, because when we need to reupload some historical data we will be ensured that there will be no data duplication because all the selected period for selected accounts will be deleted and reuploaded by the fresh data?
You can use the OVERWRITE method for the tables that don't have date field in it and are not partitioned.

[vladflaks]

That’s a great question! Both methods have their pros and cons.

MERGE:
✅ Idempotency ensured on the Google BigQuery side
✅ No need to create and manage external files
❌ Extra cost for data processing
❌ Query size limit must be considered

APPEND/DELETE:
✅ No Google BigQuery processing cost
❌ Due to Apps Script limitations, ensuring idempotency is nearly impossible

Based on our experience developing proprietary data pipelines at OWOX, MERGE costs approximately $1/month per organization on average. Yes, it requires properly setting up partitioning and clustering — but you should do that anyway to minimize reporting processing costs.

So, I lean towards the MERGE option.

[vladflaks]

In Progress now

[nishantgz]

Hey Vlad, Thanks for dropping the email. Appreciate your efforts. However it seems till now all the connectors are pertaining to only Meta and Facebook Ads. Could you let me know when would be it ready for Big Query and Google Analytics? Regards <https://calendly.com/seobynish/30min> *Nishant Mehta* M: +91 8097082190 M: +91 7977960594 *Disclaimer:* This email and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by replying to this email and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this email is strictly prohibited and may be unlawful. 1. 1. 2.

…

On Sun, Mar 23, 2025 at 2:22 AM Vlad Flaks ***@***.***> wrote: In Progress <https://github.com/OWOX/js-data-connectors/tree/65-add-support-of-google-bigquery-as-data-storage> now — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPU7DVO7XFJOCLFLTNHA6AT2VXERPAVCNFSM6AAAAABWO3OR26VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENJYHA3TGNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

-- <https://www.linkedin.com/posts/gozoop_excited-to-share-that-hawk-gozoop-group-activity-7292423924351430657-yxzc?utm_source=share&utm_medium=member_ios&rcm=ACoAAChbk-EBcfxAR_QGGfxDtxTGtmPMlocLcZw> Disclaimer: This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by replying to this e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail strictly prohibited and may be unlawful.

[vladflaks]

@nishantgz
Google BigQuery support is now live — you can set it as a storage for all available data connectors (including, but not limited to, Meta Ads).
As for Google Analytics 4, you can set up RAW data export as well as scheduled reports export.

Unless Google BigQuery is your storage of choice, I don’t believe a custom connector would bring much additional value.
Feel free to share your use case if you think otherwise — we’d be happy to take a closer look.