Merge branch 'main' into serverless_video_notebook

Author: jeffvestal

.github/workflows/pre-commit.yml

@@ -9,6 +9,8 @@ jobs:
   pre-commit:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-python@v3
-    - uses: pre-commit/[email protected]
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v3
+        with:
+          python-version: '3.13'  # or '3.12'
+      - uses: pre-commit/[email protected]

bin/find-notebooks-to-test.sh

@@ -5,6 +5,7 @@ EXEMPT_NOTEBOOKS=(
     "notebooks/esql/esql-getting-started.ipynb"
     "notebooks/search/07-inference.ipynb"
     "notebooks/search/08-learning-to-rank.ipynb"
+    "notebooks/search/09-semantic-text.ipynb"
     "notebooks/search/10-semantic-reranking-retriever-cohere.ipynb"
     "notebooks/search/11-semantic-reranking-hugging-face.ipynb"
     "notebooks/search/12-semantic-reranking-elastic-rerank.ipynb"
@@ -57,7 +58,6 @@ EXEMPT_NOTEBOOKS__8_12=(
 
 EXEMPT_NOTEBOOKS__8_14=(
     # Add any notebooks that must be skipped on versions 8.14 or older here
-    "notebooks/search/09-semantic-text.ipynb",
     # This notebook has the text_expansion deprecation notice for 8.15. 
     # Only running on 8.15 so includes the deprecation notice and newer so the local output is the same as CI
     "notebooks/langchain/langchain-vector-store-using-elser.ipynb",

notebooks/enterprise-search/app-search-engine-exporter.ipynb

@@ -24,7 +24,7 @@
     "\n",
     "## Setup\n",
     "\n",
-    "Let's start by making sure our Elasticsearch and Enterprise Search clients are installed. We'll also use `getpass` to ensure we can allow secure user inputs for our IDs and keys to access our Elasticsearch instance.\n"
+    "Let's start by making sure our Elasticsearch and Enterprise Search clients are installed. We'll also use `getpass` to ensure we can allow secure user inputs for our IDs and keys to access our Elasticsearch instance. Note that we use the `elasticsearch8` package, as Enterprise Search is not available in Elasticsearch 9 and higher.\n"
    ]
   },
   {
@@ -42,12 +42,12 @@
     "# install packages\n",
     "import sys\n",
     "\n",
-    "!{sys.executable} -m pip install -qU elasticsearch elastic-enterprise-search\n",
+    "!{sys.executable} -m pip install -qU elasticsearch8 elastic-enterprise-search\n",
     "\n",
     "# import modules\n",
     "from getpass import getpass\n",
     "from elastic_enterprise_search import AppSearch\n",
-    "from elasticsearch import Elasticsearch\n",
+    "from elasticsearch8 import Elasticsearch\n",
     "import json"
    ]
   },
@@ -335,7 +335,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -369,7 +369,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -425,7 +425,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -515,7 +515,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -545,7 +545,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -675,7 +675,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -741,7 +741,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -770,7 +770,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [

notebooks/langchain/multi-query-retriever-examples/langchain-multi-query-retriever.ipynb

@@ -38,7 +38,7 @@
     }
    ],
    "source": [
-    "!python3 -m pip install -qU lark langchain langchain-elasticsearh langchain_openai tiktoken\n",
+    "!python3 -m pip install -qU lark langchain langchain-elasticsearch langchain_openai tiktoken\n",
     "\n",
     "from langchain.schema import Document\n",
     "from langchain_openai.embeddings import OpenAIEmbeddings\n",

notebooks/search/09-semantic-text.ipynb

@@ -34,7 +34,7 @@
     "- An Elastic deployment:\n",
     "  - We'll be using [Elastic Cloud](https://www.elastic.co/guide/en/cloud/current/ec-getting-started.html) for this example (available with a [free trial](https://cloud.elastic.co/registration?onboarding_token=vectorsearch&utm_source=github&utm_content=elasticsearch-labs-notebook))\n",
     "\n",
-    "- Elasticsearch 8.15 or above, or [Elasticsearch serverless](https://www.elastic.co/elasticsearch/serverless)"
+    "- Elasticsearch 9.1 or above, or [Elasticsearch serverless](https://www.elastic.co/elasticsearch/serverless)"
    ]
   },
   {
@@ -84,7 +84,7 @@
    },
    "outputs": [],
    "source": [
-    "!pip install \"elasticsearch<9\""
+    "!pip install elasticsearch"
    ]
   },
   {
@@ -241,109 +241,6 @@
     "Read [this page](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new) to learn how to connect using API keys."
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "22fa643780acd44a",
-   "metadata": {
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
-   },
-   "source": [
-    "## Create the Inference Endpoint\n",
-    "\n",
-    "Let's create the inference endpoint by using the [Create inference API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html).\n",
-    "\n",
-    "For this example we'll use the [ELSER service](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html), but the inference API also supports [many other inference services](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html#put-inference-api-desc)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "8ee2188ea71324f5",
-   "metadata": {
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
-   },
-   "outputs": [],
-   "source": [
-    "try:\n",
-    "    client.inference.delete(inference_id=\"my-elser-endpoint\")\n",
-    "except exceptions.NotFoundError:\n",
-    "    # Inference endpoint does not exist\n",
-    "    pass\n",
-    "\n",
-    "try:\n",
-    "    client.options(\n",
-    "        request_timeout=60, max_retries=3, retry_on_timeout=True\n",
-    "    ).inference.put(\n",
-    "        task_type=\"sparse_embedding\",\n",
-    "        inference_id=\"my-elser-endpoint\",\n",
-    "        body={\n",
-    "            \"service\": \"elser\",\n",
-    "            \"service_settings\": {\"num_allocations\": 1, \"num_threads\": 1},\n",
-    "        },\n",
-    "    )\n",
-    "    print(\"Inference endpoint created successfully\")\n",
-    "except exceptions.BadRequestError as e:\n",
-    "    if e.error == \"resource_already_exists_exception\":\n",
-    "        print(\"Inference endpoint created successfully\")\n",
-    "    else:\n",
-    "        raise e"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e94fd66761fd8087",
-   "metadata": {
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
-   },
-   "source": [
-    "Once the endpoint is created, we must wait until the backing ELSER service is deployed.\n",
-    "This can take a few minutes to complete."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "adb33329ce20b2f1",
-   "metadata": {
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
-   },
-   "outputs": [],
-   "source": [
-    "inference_endpoint_info = client.inference.get(inference_id=\"my-elser-endpoint\")\n",
-    "model_id = inference_endpoint_info[\"endpoints\"][0][\"service_settings\"][\"model_id\"]\n",
-    "\n",
-    "while True:\n",
-    "    status = client.ml.get_trained_models_stats(\n",
-    "        model_id=model_id,\n",
-    "    )\n",
-    "\n",
-    "    deployment_stats = status[\"trained_model_stats\"][0].get(\"deployment_stats\")\n",
-    "    if deployment_stats is None:\n",
-    "        print(\"ELSER Model is currently being deployed.\")\n",
-    "        time.sleep(5)\n",
-    "        continue\n",
-    "\n",
-    "    nodes = deployment_stats.get(\"nodes\")\n",
-    "    if nodes is not None and len(nodes) > 0:\n",
-    "        print(\"ELSER Model has been successfully deployed.\")\n",
-    "        break\n",
-    "    else:\n",
-    "        print(\"ELSER Model is currently being deployed.\")\n",
-    "    time.sleep(5)"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "818f7a72a83b5776",
@@ -356,7 +253,8 @@
    "source": [
     "## Create the Index\n",
     "\n",
-    "Now we need to create an index with a `semantic_text` field. Let's create one that enables us to perform semantic search on movie plots."
+    "Now we need to create an index with a `semantic_text` field. Let's create one that enables us to perform semantic search on movie plots.\n",
+    "We use the preconfigured `.elser-2-elastic` inference endpoint which uses the ELSER model via the [Elastic Inference Service](https://www.elastic.co/docs/explore-analyze/elastic-inference/eis)."
    ]
   },
   {
@@ -381,7 +279,7 @@
     "            \"plot\": {\"type\": \"text\", \"copy_to\": \"plot_semantic\"},\n",
     "            \"plot_semantic\": {\n",
     "                \"type\": \"semantic_text\",\n",
-    "                \"inference_id\": \"my-elser-endpoint\",\n",
+    "                \"inference_id\": \".elser-v2-elastic\",\n",
     "            },\n",
     "        }\n",
     "    },\n",

supporting-blog-content/agent-builder-a2a-agent-framework/Dockerfile

@@ -0,0 +1,15 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+RUN python -m venv /opt/venv
+
+ENV PATH="/opt/venv/bin:$PATH"
+
+COPY requirements.txt .
+
+RUN pip install -r requirements.txt
+
+COPY elastic_agent_builder_a2a.py .
+
+CMD ["python", "elastic_agent_builder_a2a.py"]