Avoid to use the hardcoded spark version in the python/delta/pip_utils.py

[openinx]

Opened a separate issue for this comment: #5798 (comment)

We need to avoid to use the hardcoded spark version for the python/delta/pip_utils.py : https://github.com/delta-io/delta/blob/master/python/delta/pip_utils.py#L83-L86

[AnudeepKonaboina]

@openinx -- i will work on this and make it automated

[openinx]

Hi @AnudeepKonaboina , seems like @tdas is working on this small issue in the way to align the spark release artifacts name. #5840 (comment)

Please take a look for that one. Thank you for your attention.

[AnudeepKonaboina]

Hey @openinx ,

I have made this change to get the right artifact name .. I believe this would always work of any version

import pyspark
    spark_version = pyspark.__version__

    scala_version = "2.13"

    # Determine the default Spark version for this Delta Python package from wheel metadata.
    #
    # For the "default" Spark line, Delta publishes an un-versioned artifact:
    #   - io.delta:delta-spark_<scala>:<deltaVersion> (e.g. delta-spark_2.13)
    #
    # For other supported Spark lines, Delta publishes Spark-versioned artifacts:
    #   - io.delta:delta-spark_<sparkMajorMinor>_<scala>:<deltaVersion> (e.g. delta-spark_4.0_2.13)
    #
    # We infer the "default" Spark line by parsing the installed `delta-spark` requirement on
    # PySpark (declared in the package metadata), instead of hardcoding a Spark version here.

    def _default_spark_version_prefix_from_delta_spark() -> Optional[str]:
        import re

        try:
            reqs = importlib_metadata.requires("delta_spark") or []
        except Exception:
            reqs = []

        for req in reqs:
            # Example: "pyspark>=3.5.2,<3.6.0" or "pyspark>=4.0.0"
            if not isinstance(req, str) or not req.lower().startswith("pyspark"):
                continue
            lower = re.search(r">=\s*([0-9]+)\.([0-9]+)", req)
            if not lower:
                continue

            major = int(lower.group(1))
            minor = int(lower.group(2))

            # If the requirement is pinned to a specific minor line via an upper bound like
            # "<3.6.0" (for pyspark>=3.5.x), treat "3.5" as the default line.
            upper = re.search(r"<\s*([0-9]+)\.([0-9]+)", req)
            if upper:
                upper_major = int(upper.group(1))
                upper_minor = int(upper.group(2))
                if upper_major == major and upper_minor == minor + 1:
                    return f"{major}.{minor}."

            # Otherwise (e.g. "pyspark>=4.0.0"), allow all minors within the major line.
            return f"{major}."
        return None

    default_spark_version_prefix = _default_spark_version_prefix_from_delta_spark()

    artifact_name = f"delta-spark_{scala_version}"

    if spark_version:
        spark_major_minor = ".".join(spark_version.split(".")[:2])  # e.g., "3.5" or "4.0"
        if default_spark_version_prefix is None:
            # Best-effort fallback: treat the running Spark as "default" so we use the
            # un-versioned artifact, which should always exist.
            default_spark_version_prefix = f"{spark_major_minor}."

        # If not the default Spark version, add Spark version to artifact name
        if not spark_version.startswith(default_spark_version_prefix):
            artifact_name = f"delta-spark_{spark_major_minor}_{scala_version}"

    maven_artifact = f"io.delta:{artifact_name}:{delta_version}"

    extra_packages = extra_packages if extra_packages is not None else []
    all_artifacts = [maven_artifact] + extra_packages
    packages_str = ",".join(all_artifacts)

    return spark_session_builder.config("spark.jars.packages", packages_str)

• Read Delta’s PySpark dependency metadata to infer which Spark version family Delta is primarily built for (the default Spark line).
• If the dependency pins a single minor line (e.g. >=3.5,<3.6), treat that minor as default; otherwise use the major line (e.g. >=4.0).
• Get the running Spark version from pyspark.__version__.
• If the running Spark version belongs to the default Spark line, use the unversioned artifact delta-spark_<scala>.
• Otherwise, use the Spark-versioned artifact delta-spark_<sparkMajorMinor>_<scala>.

cc; @tdas