feat: Support schemas in the migration tool (#4236)

- Add schema support to the migration tool.

TODO
- Add an acceptance test.
- Rename the handleIf func

Author: sfc-gh-jmichalak

pkg/scripts/migration_script/CONTRIBUTING.md

@@ -19,7 +19,7 @@ We would appreciate that any planned changes are discussed first (e.g., in an is
 
 If you want to add a new object type, you need to define it in the migration script.
 The [`program.go`](./program.go) file contains the main logic for the user interactions with the migration script through the terminal.
-At the top of the file, you need to add a new constant for the object type. Remember to reflect this change 
+At the top of the file, you need to add a new constant for the object type. Remember to reflect this change
 in the help text (in `parseInputArguments` method for the Program struct) and the readme file ([syntax section](./README.md#syntax)).
 
 Next, you can use the newly defined object type in the `generateOutput` Program method.
@@ -65,13 +65,13 @@ type databaseRow struct {
 Under schema definition, you need to provide a conversion function that takes the CSV schema struct as input and returns the corresponding SDK object.
 In our case, it would be `func (row databaseRow) convert() (*sdk.Database, error)`.
 
-> The SDK's databaseRow has the same convert method. 
+> The SDK's databaseRow has the same convert method.
 > It's worth checking as it may contain the necessary parts for converting Snowflake output for certain cases.
 
 ## 3. Providing an object migration function
 
 Now, you need to provide a function that would take the CSV input and return the generated resources and imports in the form of string (see `HandleGrants` function).
-The file with mapping function should be placed in the file named `<object_type>_migration.go`, where `<object_type>` is the name of the object type you are working with.
+The file with mapping function should be placed in the file named `mappings_<object_type>.go`, where `<object_type>` is the name of the object type you are working with.
 
 At the top of the function, you need to parse the CSV input into the CSV schema struct you have defined in the previous step.
 This is done by the predefined `ConvertCsvInput` function (see `HandleGrants` for example usage).
@@ -81,7 +81,7 @@ To do this, you should use the model package in our project that contains the lo
 It should contain the resource model struct and functions for transforming the model (if not you should add them, look at [generators documentation](https://github.com/snowflakedb/terraform-provider-snowflake/blob/main/pkg/acceptance/bettertestspoc/README.md)).
 They in combination with `TransformResourceModel` function produce the final resource definitions output.
 
-To generate the import statements or blocks, you should use the `TransformImportModel` function that expects you 
+To generate the import statements or blocks, you should use the `TransformImportModel` function that expects you
 to provide the import model with resource address and identifier used for importing a given object.
 You should look at given resource documentation to understand how to construct the resource import (e.g., https://registry.terraform.io/providers/snowflakedb/snowflake/latest/docs/resources/database#import).
 

pkg/scripts/migration_script/README.md

@@ -69,6 +69,12 @@ where script options are:
     Limitations:
       - grants on 'future' or on 'all' objects are not supported
       - all_privileges and always_apply fields are not supported
+  - `schemas` which expects a converted CSV output from the snowflake_schemas data source
+    To support object parameters, one should use the SHOW PARAMETERS output, and combine it with the SHOW SCHEMA output, so the CSV header looks like `"comment","created_on",...,"catalog_value","catalog_level","data_retention_time_in_days_value","data_retention_time_in_days_level",...`
+    When the additional columns are present, the resulting resource will have the parameters values, if the parameter level is set to "SCHEMA".
+    For more details about using multiple sources, visit the [Multiple sources section](#multiple-sources).
+    Supported resources:
+      - snowflake_schema
 - **INPUT**:
   - Migration script operates on STDIN input in CSV format. You can redirect the input from a file or pipe it from another command.
 - **OUTPUT**:
@@ -288,7 +294,7 @@ No changes. Your infrastructure matches the configuration.
 #### 5. Update generated resources
 
 The last step is optional, but highly recommended. The generated resources have generic names, which are not very user-friendly,
-and they do not depend on the existing role resources which they refer to in their configuration. 
+and they do not depend on the existing role resources which they refer to in their configuration.
 
 To rename the resources, you can use the [terraform state mv](https://developer.hashicorp.com/terraform/cli/commands/state/mv) command or [moved block](https://developer.hashicorp.com/terraform/language/moved).
 
@@ -377,7 +383,7 @@ SHOW GRANTS ON ACCOUNT;
 and filter the output to only include the grants to the roles we are interested in.
 
 > If you use SnowSight, you can click on the "Download" button and select "CSV" format.
-> 
+>
 > ![Download CSV button in SnowSight](./images/csv_output_download.png)
 
 Whatever way you choose, save the output to a CSV file as `example.csv`.
@@ -526,6 +532,84 @@ Remember that, if you chose to use the import block approach, [after importing y
 
 By following the above steps, you can migrate other existing Snowflake objects into Terraform and start managing them!
 
+### Multiple sources
+Some Snowflake objects (like schemas) have fields returned from more than one SQL command. That's why simply using one `SHOW ...` output will not work. Fields from `DESCRIBE` or `SHOW PARAMETERS` must be also processed.
+But outputs from all of these commands must be mapped to the input CSV value of the migration script. To make this easy, we can use a data source output for a given object, which already has the logic for mapping multiple
+SQL queries and returning all necessary information.
+
+In general, what we need to do is:
+1. Define a data source for the objects you want to import.
+1. Use HCL (Terraform's configuration language) to transform the data: merge `show_output` with flattened `parameters` for each object.
+1. Write the transformed data to a CSV file using the `local_file` resource.
+1. Run the migration script with the generated CSV file.
+
+> **Note:** It's recommended to create a fresh Terraform environment (e.g., a new local directory with a clean state) for this data extraction process to avoid collisions with any existing data sources or state in your main Terraform workspace.
+
+Now, let's look into more details.
+
+As an example, let's import all schemas in a given database. First, we need to define a data source for schemas and use Terraform's HCL to transform the data into CSV format:
+
+```terraform
+terraform {
+  required_providers {
+    snowflake = {
+      source = "Snowflake-Labs/snowflake"
+    }
+    local = {
+      source = "hashicorp/local"
+    }
+  }
+}
+
+data "snowflake_schemas" "test" {
+  in {
+    database = "DATABASE"
+  }
+}
+
+locals {
+  # Transform each schema by merging show_output and flattened parameters
+  schemas_flattened = [
+    for schema in data.snowflake_schemas.test.schemas : merge(
+      schema.show_output[0],
+      # Flatten parameters: convert each parameter to {param_name}_value and {param_name}_level
+      {
+        for param_key, param_values in schema.parameters[0] :
+        param_key => param_values[0].value
+      },
+      {
+        for param_key, param_values in schema.parameters[0] :
+        "${param_key}_level" => param_values[0].level
+      }
+    )
+  ]
+
+  # Get all unique keys from the first schema to create CSV header
+  csv_header = join(",", [for key in keys(local.schemas_flattened[0]) : "\"${key}\""])
+
+  # Convert each schema object to CSV row
+  csv_rows = [
+    for schema in local.schemas_flattened :
+      join(",", [for key in keys(local.schemas_flattened[0]) : "\"${lookup(schema, key, "")}\""])
+  ]
+
+  # Combine header and rows
+  csv_content = join("\n", concat([local.csv_header], local.csv_rows))
+}
+
+resource "local_file" "schemas_csv" {
+  content  = local.csv_content
+  filename = "${path.module}/schemas.csv"
+}
+```
+
+After running `terraform apply`, the CSV file will be automatically generated at `schemas.csv`. Now, we can run the migration script like:
+```shell
+go run github.com/Snowflake-Labs/terraform-provider-snowflake/pkg/scripts/migration_script@main -import=block schemas < ./schemas.csv
+```
+
+This will output the generated configuration and import blocks for the specified schemas.
+
 ## Limitations
 
 ### Generated resource names

pkg/scripts/migration_script/common.go

@@ -0,0 +1,68 @@
+package main
+
+import (
+	"strconv"
+
+	"github.com/Snowflake-Labs/terraform-provider-snowflake/pkg/acceptance/bettertestspoc/config/model"
+	"github.com/Snowflake-Labs/terraform-provider-snowflake/pkg/sdk"
+)
+
+func handleOptionalFieldWithBuilder[T any](parameter *T, builder func(T) *model.SchemaModel) {
+	if parameter != nil {
+		builder(*parameter)
+	}
+}
+
+func handleIfNotEmpty(value string, builder func(string) *model.SchemaModel) {
+	if value != "" {
+		builder(value)
+	}
+}
+
+func handleIf(condition bool, builder func(string) *model.SchemaModel) {
+	if condition {
+		builder("true")
+	}
+}
+
+type parameterHandler struct {
+	level sdk.ParameterType
+}
+
+func newParameterHandler(level sdk.ParameterType) parameterHandler {
+	return parameterHandler{
+		level: level,
+	}
+}
+
+func (h *parameterHandler) handleIntegerParameter(level sdk.ParameterType, value string, setField **int) error {
+	if h.level != level {
+		return nil
+	}
+	v, err := strconv.Atoi(value)
+	if err != nil {
+		return err
+	}
+	*setField = &v
+	return nil
+}
+
+func (h *parameterHandler) handleBooleanParameter(level sdk.ParameterType, value string, setField **bool) error {
+	if h.level != level {
+		return nil
+	}
+	b, err := strconv.ParseBool(value)
+	if err != nil {
+		return err
+	}
+	*setField = &b
+	return nil
+}
+
+func (h *parameterHandler) handleStringParameter(level sdk.ParameterType, value string, setField **string) error {
+	if h.level != level {
+		return nil
+	}
+	*setField = &value
+	return nil
+}

pkg/scripts/migration_script/converter_schemas.go

@@ -0,0 +1,124 @@
+package main
+
+import (
+	"errors"
+
+	"github.com/Snowflake-Labs/terraform-provider-snowflake/pkg/sdk"
+)
+
+var _ ConvertibleCsvRow[SchemaRepresentation] = new(SchemaCsvRow)
+
+type SchemaCsvRow struct {
+	Comment                                      string `csv:"comment"`
+	CreatedOn                                    string `csv:"created_on"`
+	DatabaseName                                 string `csv:"database_name"`
+	DroppedOn                                    string `csv:"dropped_on"`
+	IsCurrent                                    string `csv:"is_current"`
+	IsDefault                                    string `csv:"is_default"`
+	Name                                         string `csv:"name"`
+	Options                                      string `csv:"options"`
+	Owner                                        string `csv:"owner"`
+	OwnerRoleType                                string `csv:"owner_role_type"`
+	RetentionTime                                string `csv:"retention_time"`
+	CatalogValue                                 string `csv:"catalog_value"`
+	CatalogLevel                                 string `csv:"catalog_level"`
+	DataRetentionTimeInDaysValue                 string `csv:"data_retention_time_in_days_value"`
+	DataRetentionTimeInDaysLevel                 string `csv:"data_retention_time_in_days_level"`
+	DefaultDdlCollationValue                     string `csv:"default_ddl_collation_value"`
+	DefaultDdlCollationLevel                     string `csv:"default_ddl_collation_level"`
+	EnableConsoleOutputValue                     string `csv:"enable_console_output_value"`
+	EnableConsoleOutputLevel                     string `csv:"enable_console_output_level"`
+	ExternalVolumeValue                          string `csv:"external_volume_value"`
+	ExternalVolumeLevel                          string `csv:"external_volume_level"`
+	LogLevelValue                                string `csv:"log_level_value"`
+	LogLevelLevel                                string `csv:"log_level_level"`
+	MaxDataExtensionTimeInDaysValue              string `csv:"max_data_extension_time_in_days_value"`
+	MaxDataExtensionTimeInDaysLevel              string `csv:"max_data_extension_time_in_days_level"`
+	PipeExecutionPausedValue                     string `csv:"pipe_execution_paused_value"`
+	PipeExecutionPausedLevel                     string `csv:"pipe_execution_paused_level"`
+	QuotedIdentifiersIgnoreCaseValue             string `csv:"quoted_identifiers_ignore_case_value"`
+	QuotedIdentifiersIgnoreCaseLevel             string `csv:"quoted_identifiers_ignore_case_level"`
+	ReplaceInvalidCharactersValue                string `csv:"replace_invalid_characters_value"`
+	ReplaceInvalidCharactersLevel                string `csv:"replace_invalid_characters_level"`
+	StorageSerializationPolicyValue              string `csv:"storage_serialization_policy_value"`
+	StorageSerializationPolicyLevel              string `csv:"storage_serialization_policy_level"`
+	SuspendTaskAfterNumFailuresValue             string `csv:"suspend_task_after_num_failures_value"`
+	SuspendTaskAfterNumFailuresLevel             string `csv:"suspend_task_after_num_failures_level"`
+	TaskAutoRetryAttemptsValue                   string `csv:"task_auto_retry_attempts_value"`
+	TaskAutoRetryAttemptsLevel                   string `csv:"task_auto_retry_attempts_level"`
+	TraceLevelValue                              string `csv:"trace_level_value"`
+	TraceLevelLevel                              string `csv:"trace_level_level"`
+	UserTaskManagedInitialWarehouseSizeValue     string `csv:"user_task_managed_initial_warehouse_size_value"`
+	UserTaskManagedInitialWarehouseSizeLevel     string `csv:"user_task_managed_initial_warehouse_size_level"`
+	UserTaskMinimumTriggerIntervalInSecondsValue string `csv:"user_task_minimum_trigger_interval_in_seconds_value"`
+	UserTaskMinimumTriggerIntervalInSecondsLevel string `csv:"user_task_minimum_trigger_interval_in_seconds_level"`
+	UserTaskTimeoutMsValue                       string `csv:"user_task_timeout_ms_value"`
+	UserTaskTimeoutMsLevel                       string `csv:"user_task_timeout_ms_level"`
+}
+
+type SchemaRepresentation struct {
+	sdk.Schema
+
+	// parameters
+	Catalog                                 *string
+	DataRetentionTimeInDays                 *int
+	DefaultDdlCollation                     *string
+	EnableConsoleOutput                     *bool
+	ExternalVolume                          *string
+	LogLevel                                *string
+	MaxDataExtensionTimeInDays              *int
+	PipeExecutionPaused                     *bool
+	QuotedIdentifiersIgnoreCase             *bool
+	ReplaceInvalidCharacters                *bool
+	StorageSerializationPolicy              *string
+	SuspendTaskAfterNumFailures             *int
+	TaskAutoRetryAttempts                   *int
+	TraceLevel                              *string
+	UserTaskManagedInitialWarehouseSize     *string
+	UserTaskMinimumTriggerIntervalInSeconds *int
+	UserTaskTimeoutMs                       *int
+}
+
+func (row SchemaCsvRow) convert() (*SchemaRepresentation, error) {
+	schemaRepresentation := &SchemaRepresentation{
+		Schema: sdk.Schema{
+			Name:          row.Name,
+			IsDefault:     row.IsDefault == "Y",
+			IsCurrent:     row.IsCurrent == "Y",
+			DatabaseName:  row.DatabaseName,
+			Owner:         row.Owner,
+			Comment:       row.Comment,
+			RetentionTime: row.RetentionTime,
+			OwnerRoleType: row.OwnerRoleType,
+		},
+	}
+	if row.Options != "" {
+		schemaRepresentation.Options = &row.Options
+	}
+
+	handler := newParameterHandler(sdk.ParameterTypeSchema)
+	errs := errors.Join(
+		handler.handleIntegerParameter(sdk.ParameterType(row.DataRetentionTimeInDaysLevel), row.DataRetentionTimeInDaysValue, &schemaRepresentation.DataRetentionTimeInDays),
+		handler.handleIntegerParameter(sdk.ParameterType(row.MaxDataExtensionTimeInDaysLevel), row.MaxDataExtensionTimeInDaysValue, &schemaRepresentation.MaxDataExtensionTimeInDays),
+		handler.handleStringParameter(sdk.ParameterType(row.ExternalVolumeLevel), row.ExternalVolumeValue, &schemaRepresentation.ExternalVolume),
+		handler.handleStringParameter(sdk.ParameterType(row.CatalogLevel), row.CatalogValue, &schemaRepresentation.Catalog),
+		handler.handleBooleanParameter(sdk.ParameterType(row.PipeExecutionPausedLevel), row.PipeExecutionPausedValue, &schemaRepresentation.PipeExecutionPaused),
+		handler.handleBooleanParameter(sdk.ParameterType(row.ReplaceInvalidCharactersLevel), row.ReplaceInvalidCharactersValue, &schemaRepresentation.ReplaceInvalidCharacters),
+		handler.handleStringParameter(sdk.ParameterType(row.DefaultDdlCollationLevel), row.DefaultDdlCollationValue, &schemaRepresentation.DefaultDdlCollation),
+		handler.handleStringParameter(sdk.ParameterType(row.StorageSerializationPolicyLevel), row.StorageSerializationPolicyValue, &schemaRepresentation.StorageSerializationPolicy),
+		handler.handleStringParameter(sdk.ParameterType(row.LogLevelLevel), row.LogLevelValue, &schemaRepresentation.LogLevel),
+		handler.handleStringParameter(sdk.ParameterType(row.TraceLevelLevel), row.TraceLevelValue, &schemaRepresentation.TraceLevel),
+		handler.handleIntegerParameter(sdk.ParameterType(row.SuspendTaskAfterNumFailuresLevel), row.SuspendTaskAfterNumFailuresValue, &schemaRepresentation.SuspendTaskAfterNumFailures),
+		handler.handleIntegerParameter(sdk.ParameterType(row.TaskAutoRetryAttemptsLevel), row.TaskAutoRetryAttemptsValue, &schemaRepresentation.TaskAutoRetryAttempts),
+		handler.handleStringParameter(sdk.ParameterType(row.UserTaskManagedInitialWarehouseSizeLevel), row.UserTaskManagedInitialWarehouseSizeValue, &schemaRepresentation.UserTaskManagedInitialWarehouseSize),
+		handler.handleIntegerParameter(sdk.ParameterType(row.UserTaskTimeoutMsLevel), row.UserTaskTimeoutMsValue, &schemaRepresentation.UserTaskTimeoutMs),
+		handler.handleIntegerParameter(sdk.ParameterType(row.UserTaskMinimumTriggerIntervalInSecondsLevel), row.UserTaskMinimumTriggerIntervalInSecondsValue, &schemaRepresentation.UserTaskMinimumTriggerIntervalInSeconds),
+		handler.handleBooleanParameter(sdk.ParameterType(row.QuotedIdentifiersIgnoreCaseLevel), row.QuotedIdentifiersIgnoreCaseValue, &schemaRepresentation.QuotedIdentifiersIgnoreCase),
+		handler.handleBooleanParameter(sdk.ParameterType(row.EnableConsoleOutputLevel), row.EnableConsoleOutputValue, &schemaRepresentation.EnableConsoleOutput),
+	)
+	if errs != nil {
+		return nil, errs
+	}
+
+	return schemaRepresentation, nil
+}