How to pass credential file while connecting to bigquery database

[jitendra-bansal]

Hi Team,
I am confuse to use data-diff for bigquery database as it is mentioned | BigQuery | bigquery://<project>/<dataset>

how to pass credential file using this?

It will be helpful if any example is added in readme to use bigquery

[erezsh]

Hi, if you want to pass a credential argument to bigquery, you can't do it via a URI. You'll have to use a config file: https://github.com/datafold/data-diff#how-to-use-with-a-configuration-file

In our code, we connect to bigquery like this:

bigquery.Client(project, **kw)

So any parameter you'll put under [database.my_bigquery] (or however you name it) will reach the Client constructor as-is.

[jitendra-bansal]

OK I will check with a configuration file.
But I am using data-diff module into my python codebase. So, how to pass this credentials using python.

like the example they have given
import logging
logging.basicConfig(level=logging.INFO)

from data_diff import connect_to_table, diff_tables

table1 = connect_to_table("postgresql:///", "table_name", "id")
table2 = connect_to_table("mysql:///", "table_name", "id")

for different_row in diff_tables(table1, table2):
plus_or_minus, columns = different_row
print(plus_or_minus, columns)

[erezsh]

@jitendra-bansal the connect_to_table() function doesn't allow it yet. We'll fix it soon.

Meanwhile, it's possible if you use data_diff.connect.connect_with_dict() instead, where the dict keys are the same as you would pass to bigquery.Client. And then you also have to create the TableSegment instance yourself.

Edit: Hold on, I'll just fix it real quick.

[erezsh]

@jitendra-bansal I created a PR for this change. You can install it with:

pip install git+https://github.com/datafold/data-diff.git@refs/pull/157/head

You would then connect using something like this:

db1 = dict( driver = "bigquery", credentials = "..." )
table1 = connect_to_table(db1, "table_name", "id")

[jitendra-bansal]

Hi @erezsh thanks for the help this PR 157 worked for me.