{"id":10435,"date":"2025-06-28T15:36:39","date_gmt":"2025-06-28T14:36:39","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=10435"},"modified":"2025-12-20T11:25:10","modified_gmt":"2025-12-20T16:25:10","slug":"ai-powered-semantic-search-for-wordpress","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/ai-powered-semantic-search-for-wordpress\/","title":{"rendered":"AI Semantic Search Project Summary – Local PC Version"},"content":{"rendered":"
<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

STOP ALL:
\nsudo pkill -f \"python3 rag_api_server.py\"; sudo pkill -f \"ollama serve\"; sudo systemctl stop apache2; sudo systemctl stop mysql;<\/p>\n

CHECK STATUS:
\nsudo ps aux | grep -E 'apache2|mysqld|ollama|rag_api_server.py' | grep -v grep
\nstevee@hplaptop:~$ curl https:\/\/127.0.0.1:11434
\nOllama is runningstevee@hplaptop:~$<\/p>\n

Reason: This command uses ps aux to list all running processes on your system. It then filters that list for common process names associated with your services (Apache, MySQL, Ollama, and your Flask server script) and excludes the grep command itself. If a line appears in the output, that process is still running. If nothing is returned, the process is truly stopped. This is more definitive for crashed\/stuck processes than just checking port listening status.<\/p>\n

(venv_ollama_rag)stevee@hplaptop:\/var\/www\/stevee@hplaptop:~$ sudo systemctl is-active ollama
\n[sudo] password for stevee:
\nactive
\nstevee@hplaptop:~$ sudo systemctl is-active apache2
\nactive
\nstevee@hplaptop:~$ sudo systemctl is-active mysql.service
\nactive<\/p>\n

START ALL:<\/p>\n

cd \/var\/www\/my-llm-project\/ ; source llm_venv\/bin\/activate ;<\/span>
\nsudo service apache2 start &>\/dev\/null && echo \"Apache2: $(sudo systemctl is-active apache2 && echo 'active' || echo 'failed')\";<\/span>
\nsudo service mysql start &>\/dev\/null && echo \"MYSQL: $(sudo systemctl is-active mysql && echo 'active' || echo 'failed')\";<\/span>
\nsudo systemctl start ollama &>\/dev\/null && echo \"ollama: $(sudo systemctl is-active ollama || echo 'failed')\" ;<\/span>
\nsudo lsof -i :11434 2>\/dev\/null | grep \"LISTEN\" ; echo \"Starting Flask server (this will occupy the terminal)...\" ;<\/span>
\n\/var\/www\/my-llm-project\/llm_venv\/bin\/python semantic_search_api.py<\/span><\/p>\n

---------- SUDO CHANGES USER PATHS PERMS!; SO FULL PATH REQUIRED FOR PYTHON ...py files<\/p>\n

OUTPUT:<\/p>\n

[sudo] password for stevee:
\nApache2: active
\nactive
\nMySQL\/MariaDB: active
\nactive
\nOllama: active
\nactive
\nStarting Flask server (this will occupy the terminal)...
\nLoaded 148 embeddings and post data.
\nGemini model initialized successfully using 'gemini-1.5-flash-latest'.
\n* Serving Flask app 'semantic_search_api'
\n* Debug mode: on
\nWARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
\n* Running on all addresses (0.0.0.0)
\n* Running on https:\/\/127.0.0.1:5000
\n* Running on https:\/\/stevepedwards.today:5000
\nPress CTRL+C to quit
\n* Restarting with stat
\nLoaded 148 embeddings and post data.
\nGemini model initialized successfully using 'gemini-1.5-flash-latest'.
\n* Debugger is active!
\n* Debugger PIN: 203-532-454
\n127.0.0.1 - - [06\/Jul\/2025 12:25:51] \"GET \/socket.io\/?EIO=4&transport=polling&t=PVWnS-D HTTP\/1.1\" 200 -
\nClient connected via WebSocket!
\n127.0.0.1 - - [06\/Jul\/2025 12:25:51] \"POST \/socket.io\/?EIO=4&transport=polling&t=PVWnTPq&sid=Qou_ABbX6pI_eGBwAAAA HTTP\/1.1\" 200 -
\n127.0.0.1 - - [06\/Jul\/2025 12:25:51] \"GET \/socket.io\/?EIO=4&transport=polling&t=PVWnTPr&sid=Qou_ABbX6pI_eGBwAAAA HTTP\/1.1\" 200 -
\n127.0.0.1 - - [06\/Jul\/2025 12:25:51] \"GET \/socket.io\/?EIO=4&transport=polling&t=PVWnTPy&sid=Qou_ABbX6pI_eGBwAAAA HTTP\/1.1\" 200 -<\/p>\n

Here's how it works in your startup script:<\/p>\n

    \n
  • sudo systemctl is-active apache2 && echo 'active' || (sudo service apache2 start && echo 'restarting\/started') || echo 'failed to start'<\/code><\/b>\n
      \n
    • sudo systemctl is-active apache2<\/code>: This command checks if Apache2 is active. If it succeeds<\/i> (Apache is active), it returns a zero exit status.<\/li>\n
    • && echo 'active'<\/code>: If is-active<\/code> succeeds, then echo 'active'<\/code> runs, and the rest of the line is skipped because the condition (Apache active) has been met.<\/li>\n
    • || (sudo service apache2 start && echo 'restarting\/started')<\/code>: If is-active<\/code> fails<\/i> (Apache is not active), then this part executes. It attempts to start Apache, and if that<\/i> succeeds, it prints 'restarting\/started'.<\/li>\n
    • || echo 'failed to start'<\/code>: If both<\/i> is-active<\/code> and<\/i> the start<\/code> command fail, then this last part executes, printing 'failed to start'.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

      So, ||<\/code> provides a way to define alternative actions or fallback options if a preceding command doesn't succeed.<\/p>\n

      Your project seamlessly integrates a local Large Language Model (LLM) with your WordPress website, enabling voice-activated semantic search and AI-powered summarization of your site's content.<\/span><\/p>\n

      General Function<\/span><\/h3>\n

      The system allows users to speak (or type) a query into a search bar on your WordPress site. Instead of a traditional keyword search, it uses a local LLM to understand the meaning<\/i> of the query. This semantic understanding allows it to find the most relevant posts and pages on your site, even if they don't contain the exact keywords. It then displays these results as clickable links, and offers an option to generate an AI-powered summary of the content using an online Gemini model. Real-time feedback on backend processes is shown in a terminal-like display on the webpage.<\/span><\/p>\n

      Stages of Construction and Script Roles<\/span><\/h3>\n

      The project's functionality is built across several interconnected stages, involving WordPress (PHP, MySQL), a Python Flask API, a local Ollama LLM, and browser-side JavaScript and CSS.<\/span><\/p>\n

      Stage 1: Content Extraction & Embedding Generation<\/b> This is the foundational step where your WordPress content is prepared for semantic understanding.<\/span><\/p>\n

        \n
      • Trigger:<\/b> A new Post or Page is added\/updated on your WordPress site.<\/span><\/li>\n
      • Need for Update:<\/b> The LLM system needs to be aware of this new content to include it in future semantic searches.<\/span><\/li>\n
      • Script:<\/b> generate_embeddings.py<\/code><\/span>\n
          \n
        • Role:<\/b> This Python script is responsible for:<\/span>\n
            \n
          1. Connecting to WordPress's MySQL Database:<\/b> It queries your wp_xubg_posts<\/code> table to fetch the ID<\/code>, post_title<\/code>, post_content<\/code>, post_type<\/code>, and post_name<\/code> (slug) of all published posts and pages.<\/span><\/li>\n
          2. Cleaning Content:<\/b> It removes HTML tags and WordPress shortcodes from the raw post_content<\/code> to get clean text suitable for the LLM. It also decodes HTML entities.<\/span><\/li>\n
          3. Saving Full Post Texts:<\/b> For each cleaned post, it saves the full text into individual .txt<\/code> files in the full_posts_text\/<\/code> directory (e.g., 12345.txt<\/code>). Before doing so, it clears the entire full_posts_text\/<\/code> directory<\/b> to ensure only current and published posts are stored, preventing stale data buildup.<\/span><\/li>\n
          4. Generating Embeddings:<\/b> For each cleaned post's text, it sends the text to your locally running Ollama server<\/b>, specifically using the nomic-embed-text<\/code> model. Ollama returns a numerical vector (embedding) that represents the semantic meaning of the text.<\/span><\/li>\n
          5. Storing Embeddings & Metadata:<\/b> It compiles all this information (Post ID, Title, Slug, Content Snippet, Word Count, and the generated Embedding) into a single JSON file: wordpress_embeddings.json<\/code>.<\/span><\/li>\n<\/ol>\n<\/li>\n
          6. Why it runs:<\/b> To create or refresh the \"knowledge base\" (the embeddings) that your semantic search API will use. This must be done whenever WordPress content changes to keep the search accurate.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

            Stage 2: Backend API Server (Semantic Search & Summarization)<\/b> This stage provides the intelligent services that the WordPress frontend will communicate with.<\/span><\/p>\n

              \n
            • Trigger:<\/b> Your Flask server needs to be running in the background. It needs to be restarted whenever wordpress_embeddings.json<\/code> is updated or semantic_search_api.py<\/code> itself is changed.<\/span><\/li>\n
            • Script:<\/b> semantic_search_api.py<\/code><\/span>\n
                \n
              • Role:<\/b> This Python Flask application acts as the bridge between your WordPress frontend and the LLM services.<\/span>\n
                  \n
                1. Loads Knowledge Base:<\/b> On startup, it reads the wordpress_embeddings.json<\/code> file into memory. It stores all the post metadata and their corresponding embeddings.<\/span><\/li>\n
                2. Initializes Ollama Connection:<\/b> Sets up the connection to your local Ollama server to handle embedding requests for search queries.<\/span><\/li>\n
                3. Initializes Gemini Connection:<\/b> Configures the connection to the Google Gemini API, specifically using the models\/gemini-1.5-flash-latest<\/code> model for summarization.<\/span><\/li>\n
                4. Semantic Search Endpoint (\/semantic_search<\/code>):<\/b> When a search query is received from the frontend:<\/span>\n
                    \n
                  • It sends the user's query to Ollama (using nomic-embed-text<\/code>) to generate an embedding for the query.<\/span><\/li>\n
                  • It compares this query embedding to all the stored embeddings from your WordPress posts using cosine similarity.<\/span><\/li>\n
                  • It identifies the top 3 most semantically similar posts from your site.<\/span><\/li>\n
                  • It returns the metadata (Title, Slug, Word Count, Score, Content Snippet) of these top 3 results to the frontend.<\/span><\/li>\n<\/ul>\n<\/li>\n
                  • Summarization Endpoint (\/summarize_article<\/code>):<\/b> When the \"Summarise Article\" button is clicked for a specific post:<\/span>\n
                      \n
                    • It reads the full content of that post from the .txt<\/code> file in full_posts_text\/<\/code>.<\/span><\/li>\n
                    • It sends this full content as a prompt to the online Google Gemini API<\/b> (gemini-1.5-flash-latest<\/code>) to generate a concise summary.<\/span><\/li>\n
                    • It returns the generated summary back to the frontend.<\/span><\/li>\n<\/ul>\n<\/li>\n
                    • Real-time Communication:<\/b> Uses Socket.IO to send real-time progress updates from the backend Python scripts directly to the terminal display on your WordPress page.<\/span><\/li>\n<\/ol>\n<\/li>\n
                    • Why it runs:<\/b> It continuously listens for requests from your WordPress site and provides the semantic search and summarization intelligence.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                      Stage 3: Frontend User Interface & Interaction<\/b> This stage handles how users interact with the system on your WordPress site.<\/span><\/p>\n

                        \n
                      • Trigger:<\/b> User visits your WordPress page.<\/span><\/li>\n
                      • Scripts:<\/b><\/span>\n
                          \n
                        • my_voice_search.php<\/code> (WordPress Plugin PHP)<\/span>\n
                            \n
                          • Role:<\/b> This WordPress plugin handles the display of the search interface.<\/span>\n
                              \n
                            1. Enqueues Assets:<\/b> It loads your style.css<\/code> (for styling), socket.io.min.js<\/code> (for real-time updates), and speech-recognition.js<\/code> (for logic) into your WordPress page.<\/span><\/li>\n
                            2. Registers Shortcodes:<\/b> It defines two shortcodes:<\/span>\n
                                \n