SNOW-1892865: Starting an query in AsyncMode and then evicting the connection pool cancels the async query

[john-skyward-solutions]

We've found that starting an async query and then evicting the connection leads to session, and the queries still running on that session, to be cancelled.

I understand why that makes sense in a synchronous world, but in an async world where someone has deliberately started a query in async mode to leave it running it makes little sense.

Is there a way we can either force a new session to begin on the existing open connection after the async mode is started OR evict the connection from the pool in such a way that it does not stop async queries and it's transactions it has open?

Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!

1. What version of .NET driver are you using?
   4.2.0

2. What operating system and processor architecture are you using?
   All of Windows x64 / macOS silicon / Linux x64

3. What version of .NET framework are you using?
   dotnet 8.0

4. What did you do?

`using var conn = await new SnowflakeDbConnection("");
conn.Open();
conn.PreventPooling();

var cmd = conn.CreateSnowflakeDbCommand();
cmd.CommandText = "<async query that takes over 5 minutes>";
var queryId = await cmd.ExecuteAsyncInAsyncMode(default);

Console.WriteLine("QueryID started in async mode: {0}", queryId);`

5 minutes after running the above the query will be cancelled.

5. What did you expect to see?
   The query should continue to execute until completion.

[sfc-gh-dszmolka]

hi - Happy New Year :)

The behaviour you described -at the first superficial glance- does not seem to be related to this library, but the Snowflake backend itself. Especially the 5 minutes you mentioned is suspicious.

Please see the documentation for ABORT_DETACHED_QUERY - isn't this what we're seeing here? Can you please verify if the parameter is set to FALSE in case in these queries? If it's TRUE then it's expected to see the behaviour you're reporting, regardless of programming language or client driver type.

[john-skyward-solutions]

Yes we set ABORT_DETACHED_QUERY to FALSE before all Async Queries because we run into the cancelled issue for longer running queries. This is why I was so susprised to see this behaviour when preventing the connection to be pooled, because we already set it to false.

The below screenshot is every query in a single session, where we set abort detached query to false and then run our async query after that. You'll see that it still fails.

When a connection is removed from the pool, is the session aborted or similar?

[sfc-gh-dszmolka]

yes connection Close() (and Dispose() too) actively closed the underlying Snowflake Session when it's called.

it would be interesting to see the queryId of one of the failed queries but i'm almost sure the Snowflake engine (so not this library in which the issue was filed) aborts it due to being disconnected. Can you share at least one of them ?

speaking about the libraries; it would be interestin to run the same query with another Snowflake driver which supports async mode (JDBC driver, Python driver, now Node.js driver too i believe) and compare the behaviour. I would assume it's the same.

[john-skyward-solutions]

@sfc-gh-dszmolka I'm afraid I can't, but it's very easy to replicate on yourside by following the same approach.

  // create connection
  conn.Open();
  conn.PreventPooling();

  var cmd = conn.CreateCommand();

  cmd.CommandText = "alter session set abort_detached_query = false;";
  await cmd.ExecuteNonQueryAsync();

  cmd.CommandText = "<long running sql query text>";
  var queryId = await cmd.ExecuteAsyncInAsyncMode(default);

  Console.WriteLine("QueryID started in async mode: {0}", queryId);

Then watch that query to see it stop after 5 minutes.

I'd wondered the same on the other libraries. I wasn't sure if they didn't have connection pooling, so the problem did not exist, but it looks like they do.

I suppose another wider concern is that if an application shuts down fully, and any connections in the pool are closed, that would stop any async queries from completing?

[sfc-gh-dszmolka]

thank you for this example. also after all i was able to find the session and all queries run within it for the above attempt. Snowflake Cloud Services logs clearly indicate that the query was aborted on explicit abort request from the user, and not some timeout or such.

sp based on the code you shared and our documentation on how to run async queries i attempted the following:

using Snowflake.Data.Client;

const string connectionString = "account=myaccount.west-europe.azure;user=myuser;password=mypassword;insecuremode=true";
await using var conn = new SnowflakeDbConnection(connectionString);

var sleepAmount = 420;
var sleepTime = TimeSpan.FromSeconds(sleepAmount);

conn.Open();
conn.PreventPooling();
SnowflakeDbCommand cmd = (SnowflakeDbCommand)conn.CreateCommand();

cmd.CommandText = "alter session set abort_detached_query = false;";
await cmd.ExecuteNonQueryAsync();

string thisWontFinishInFive = @"SELECT * FROM
 SNOWFLAKE_SAMPLE_DATA.TPCH_SF1000.CUSTOMER
,SNOWFLAKE_SAMPLE_DATA.TPCH_SF1000.LINEITEM
,SNOWFLAKE_SAMPLE_DATA.TPCH_SF1000.ORDERS
";

cmd.CommandText = thisWontFinishInFive;
var queryId = await cmd.ExecuteAsyncInAsyncMode(CancellationToken.None).ConfigureAwait(false);

Console.WriteLine("QueryID started in async mode: {0}", queryId);
Console.WriteLine("Now sleeping for {0} seconds.", sleepAmount);
Thread.Sleep(sleepTime);
Console.WriteLine("Waking up and checking on status of query {0}", queryId);
var queryStatus = await cmd.GetQueryStatusAsync(queryId, CancellationToken.None).ConfigureAwait(false);
var isQueryRunning = conn.IsStillRunning(queryStatus);
Console.WriteLine("Is query still running? {0}", isQueryRunning);

result:

QueryID started in async mode: 01b9c0ad-0305-5817-0003-351200031022
Now sleeping for 420 seconds.
Waking up and checking on status of query 01b9c0ad-0305-5817-0003-351200031022
Is query still running? True

when checking Query History on the GUI, the query was indeed still in Running even after 7 minutes. I had to manually SYSTEM$CANCEL_QUERY it.

I believe the biggest difference might be how you are using the cancellation token in ExecuteAsyncInAsyncMode; can you please try something shared above and see if it helps keeping async queries alive past the 5m mark when ABORT_DETACHED_QUERY=false?