Using Flameshot to find all similar pictures on computer

[gnusupport]

Here is an idea that works well on my computer, utilizing the power of new Large Language Models (LLMs), or so-called artificial intelligence, along with Flameshot to crop and find similar pictures on the computer.

See here video how it works: https://www.youtube.com/watch?v=axjuTzF6FqM

There is "AI" used, that i am using nomic-ai/nomic-embed-vision-v1.5 model that runs always in memory, and if someone needs help running it, feel free to ask.
https://huggingface.co/nomic-ai/nomic-embed-vision-v1.5/

There is also a use of PostgreSQL database to index the embeddings of those files and images. Think of an embedding as a way to turn words, phrases, or sentences into numbers – like a secret code that computers can understand. Each image is indexed in the database with its embedding. I use Emacs Lisp for that.

flameshot-find-similar.sh

#!/bin/bash
# flameshot-find-similar.sh
# Complete pipeline to capture screenshot, find similar images, and open results in ROX

# Run the entire pipeline
flameshot-to-image-path.sh | \
rcd-llm-get-image-embeddings.sh | \
rcd-find-similar-images.sh | \
rcd-open-images-temp-dir-pipe.sh

flameshot-to-image-path.sh:

#!/usr/bin/env bash
TARGET_DIR="/home/data1/protected/Media/Pictures/Flameshots"
IMAGE="similar-to-$(date +'%F_%H-%M-%S').png"
TARGET_PATH="$TARGET_DIR/$IMAGE"

flameshot gui -p "$TARGET_PATH" >/dev/null 2>&1

# Wait until file exists and is fully written
last=0
while true; do
    if [ -f "$TARGET_PATH" ]; then
        size=$(stat -c '%s' "$TARGET_PATH")
        [ "$size" -gt 0 ] && [ "$size" -eq "$last" ] && break
        last="$size"
    fi
    sleep 0.2
done

# Output the file path for the next command
printf '%s\n' "$TARGET_PATH"

rcd-llm-get-image-embeddings.sh:

#!/usr/bin/env bash
# rcd-get-image-embedding.sh
# Usage:
#   ./rcd-get-image-embedding.sh /path/to/image.png
#   echo "/path/to/image.png" | ./rcd-get-image-embedding.sh

set -euo pipefail

# Determine file path: argument first, fallback to stdin
if [ $# -gt 0 ]; then
    FILE="$1"
else
    read -r FILE
fi

# Verify file exists
if [ ! -f "$FILE" ]; then
    echo "Error: File does not exist: $FILE" >&2
    exit 1
fi

# Verify file is an image
if ! file "$FILE" | grep -iqE 'image|bitmap|png|jpeg|jpg'; then
    echo "Error: Not a valid image file: $FILE" >&2
    exit 1
fi

# API configuration
API_URL="http://192.168.1.68:9998/embed/image"

# Send image via multipart/form-data and extract embedding
EMBEDDING=$(curl -s -X POST "$API_URL" \
    -F "file=@$FILE" \
    -H "Accept: application/json" \
    | jq -r '.embedding')

# Output only the embedding
echo "$EMBEDDING"

rcd-find-similar-images.sh:

#!/usr/bin/env bash
# rcd-find-similar-images.sh
# Usage:
#   echo "$EMBEDDING" | ./rcd-find-similar-images.sh [similarity_threshold]

set -euo pipefail

# Read embedding from stdin
EMBEDDING=$(cat)

# Optional similarity threshold argument (default 0.7)
THRESHOLD="${1:-0.7}"

DB_NAME="rcdbusiness"
DB_USER="maddox"
DB_HOST="localhost"
DB_PORT="5432"

# Query similar active files above threshold and output file names only, one per line
psql "$DB_NAME" -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -t -A <<SQL
WITH TargetEmbedding AS (
    SELECT '$EMBEDDING'::vector AS embeddings
),
SimilarEmbeddings AS (
    SELECT
        f.files_name,
        1 - (te.embeddings <=> e.embeddings_embeddings) AS similarity
    FROM embeddings e
    JOIN files f ON e.embeddings_files = f.files_id
    CROSS JOIN TargetEmbedding te
    WHERE e.embeddings_embeddingtypes = 8
      AND f.files_active = 1
)
SELECT files_name
FROM SimilarEmbeddings
WHERE similarity >= $THRESHOLD
ORDER BY similarity DESC;
SQL

rcd-open-images-temp-dir-pipe.sh:

#!/bin/bash
# rcd-open-images-ultrafast.sh

temp_dir="${TMPDIR:-/tmp}/temp-dirs/$(date +%s)"
mkdir -p "$temp_dir"

echo "Creating symlinks in: $temp_dir"
echo "--------------------------------"

count=0
while IFS= read -r line; do
    file="${line/#\~/$HOME}"
    # Fast file existence check
    [ -f "$file" ] || continue
    
    base=$(basename "$file")
    # Simple unique name using counter
    dest="$temp_dir/${count}_$base"
    
    # Create symlink (fast)
    ln -sf "$file" "$dest" 2>/dev/null
    
    echo "✓ $base"
    ((count++))
done

if [[ $count -gt 0 ]]; then
    echo ""
    echo "Created $count symlink(s)"
    echo "Opening ROX..."
    
    # Open ROX in background
    (rox -n "$temp_dir" &) 2>/dev/null
    
    echo "$temp_dir"
else
    rmdir "$temp_dir"
    exit 1
fi

and then I have bound flameshot-find-similar.sh to Control-Mouse-Button 9 on my Redragon M908 but you can bind it as you wish, or invoke script from menu or by short cut.