Produce a reusable template repo that adds a **local Wikipedia search source** to the LangGraph “Open Deep Research” stack and ships runnable Kubernetes resources. Generate files with concrete content, parameterized via `{{variables}}`. Follow the spec exactly. ## Goal Create a minimal, reproducible template a downstream user can clone, set a few vars, and deploy: - Runs `open_deep_research` API with a Wikipedia search backend. - Hosts a local Kiwix mirror of Wikipedia ZIM files. - Performs semantic search with `txtai` and returns Markdown excerpts with citations to local Kiwix URLs. - Selects the Wikipedia tool via `search_api: WIKIPEDIA` without changing graph logic. ## Output format Do not add commentary outside files. Include all files listed below. ## Repository layout ``` open-deep-research-wiki-template/ README.md Makefile k8s/ resources.yaml kiwix.yaml docker/ Dockerfile entrypoint.sh run.sh run_fastapi.sh pyproject.toml uv.lock # generate a minimal lock; pin txtai + torch cpu langgraph.json # pass-through if needed src/open_deep_research/ __init__.py deep_researcher.py # integrates unchanged graph, no business-logic changes utils.py # adds wiki_search + get_search_tool wiring wikipedia.py # local Kiwix + txtai retrieval configuration.py # SearchAPI enum includes WIKIPEDIA prompts.py # unchanged state.py # unchanged configuration_example.env ``` ## Parameterization Use Jinja-style placeholders. Do not hardcode values. - `{{NAMESPACE}}` default: `open-deep-research` - `{{DOMAIN}}` default: `<domain>` - `{{WIKI_HOST}}` default: `wiki.{{DOMAIN}}` - `{{KIWIX_BASE}}` default: `https://{{WIKI_HOST}}` - `{{ZIM_NAME}}` default: `wikipedia_en_all_maxi_2025-08` - `{{API_IMAGE}}` default: `localhost:32000/mm_open_deep_research:latest` - `{{TLS_SECRET}}` default: `<domain>-wildcard-tls` - `{{REPLICAS}}` default: `1` - `{{API_PORT}}` default: `8123` - `{{CPU_LIMIT}}` default: `4` - `{{MEM_LIMIT}}` default: `56Gi` - `{{ZIM_HOSTPATH}}` default: `/mnt/data/zim` ## File requirements ### 1) `k8s/resources.yaml` - Deployment + Service for API. - Env for threading control: `FAISS_NUM_THREADS=1`, `OMP_NUM_THREADS=1`, `OPENBLAS_NUM_THREADS=1`, `MKL_NUM_THREADS=1`, `PYTHONFAULTHANDLER=1`. - `SERVER_MODE=fastapi`, `PORT={{API_PORT}}`. - Resource requests: cpu `100m`, mem `2Gi`. Limits from params. - Readiness/Liveness GET `/`. - Service on port 80 -> `{{API_PORT}}`. ### 2) `k8s/kiwix.yaml` - Deployment, Service, Ingress for `kiwix-serve`. - Container `ghcr.io/kiwix/kiwix-serve:latest` with args `kiwix-serve -p 8080 /data/zim/*.zim`. - `hostPath` volume at `{{ZIM_HOSTPATH}}` read-only. - Ingress `{{WIKI_HOST}}` with TLS `{{TLS_SECRET}}`. ### 3) `docker/Dockerfile` - Base: `python:3.11-slim`. - Install build deps: `ca-certificates curl git build-essential libgomp1`. - Install `uv` via official script. - Set caches to `/opt/cache` (HF, transformers, sentence-transformers, torch, pip). - Copy `pyproject.toml`, `uv.lock`, `src`, `server` (if present), scripts. - `uv sync` then export venv to PATH. - Set `XDG_CACHE_HOME` and related envs. - Prewarm by invoking `/app/src/open_deep_research/wikipedia.py` to trigger model pulls. - `ENTRYPOINT` to `entrypoint.sh`. ### 4) `docker/entrypoint.sh` - `SERVER_MODE` switch: `fastapi` -> `run_fastapi.sh`, `cli` -> `run.sh`. ### 5) `pyproject.toml` and `uv.lock` - Dependencies include: - `langgraph`, `langchain-*`, `openai`, `exa-py`, `tavily-python`, `pandas`, `fastapi`, `uvicorn`, `langserve`, `rich`, `httpx`, `beautifulsoup4==4.13.3`, `txtai==9.0.0`, `torch>=2.5,<3`. - Configure CPU-only torch source: ``` [tool.uv.sources] torch = [{ index = "pytorch-cpu" }] [[tool.uv.index]] name = "pytorch-cpu" url = "https://download.pytorch.org/whl/cpu" explicit = true ``` - Ensure `txtai` pulls `neuml/txtai-wikipedia` at runtime. ### 6) `src/open_deep_research/wikipedia.py` Implement: - `KIWIX_BASE=os.getenv("KIWIX_BASE","{{KIWIX_BASE}}")` - `ZIM_NAME=os.getenv("ZIM_NAME","{{ZIM_NAME}}")` - Force CPU: `os.environ["FAISS_DISABLE_GPU"]="1"`. - `Embeddings().load(provider="huggingface-hub", container="neuml/txtai-wikipedia")` behind an `@lru_cache` and `threading.RLock`. - `wiki_query(query, percentile_threshold=0.99, max_articles=5)` that: - Executes `txtai` SQL `similar('{query}')` with percentile filter. - For each hit: resolve `title`, fetch HTML from Kiwix (`/raw/...` then XML search fallback), strip to Markdown, drop references, collapse whitespace. - Return dicts: `{title,id,score,percentile,kiwix_url,markdown}`. ### 7) `src/open_deep_research/utils.py` Add: - `wiki_search` tool with signature `queries: List[str]` that calls `wiki_search_async`, flattens results, and formats `Search results:\n\n--- SOURCE i: <title> ---\nURL: <kiwix_url>\n\nSUMMARY:\n<markdown>\n\n---------`. - `get_search_tool` to map `SearchAPI.WIKIPEDIA` to `wiki_search` and expose it as `name="web_search"` so the rest of the graph is unchanged. ### 8) `src/open_deep_research/deep_researcher.py` - Keep the supervisor/researcher/compression logic as-is. - Ensure `get_all_tools(config)` path picks up `SearchAPI.WIKIPEDIA` via `utils.get_search_tool`. - No other semantic changes. ### 9) `src/open_deep_research/configuration.py` - Ensure `SearchAPI` enum includes `WIKIPEDIA`. - Default remains selectable via `config.configurable.search_api`. ### 10) `Makefile` Targets: - `build`: build API image tag `{{API_IMAGE}}`. - `push`: push image. - `deploy-kiwix`: apply `k8s/kiwix.yaml`. - `deploy-api`: apply `k8s/resources.yaml`. - `undeploy`: delete both. - `test-wiki`: curl `https://{{WIKI_HOST}}/` readiness. - `invoke`: sample JSON call with `search_api: wikipedia`. ### 11) `src/open_deep_research/configuration_example.env` Document: ``` KIWIX_BASE={{KIWIX_BASE}} ZIM_NAME={{ZIM_NAME}} FAISS_DISABLE_GPU=1 SERVER_MODE=fastapi PORT={{API_PORT}} ``` ## Quality gates Implement and satisfy these checks: 1. **Static**: repository `pip check` inside the built image must pass. 2. **Runtime**: container starts and `GET /` responds 200. 3. **Kiwix**: readiness on `/` returns 200 after ZIM mount. 4. **Search**: calling the graph with `search_api=wikipedia` returns at least one `SOURCE` block with `URL: https://{{WIKI_HOST}}/content/{{ZIM_NAME}}/A/...`. 5. **Isolation**: no external HTTP calls are required during research if Kiwix and ZIM exist. 6. **Idempotency**: `kubectl apply` twice yields no changes. ## Constraints - Use only the libraries listed in `pyproject.toml`. - Keep `txtai` CPU-only; do not enable GPU. - Keep tool name exposed to the graph as `web_search` to avoid changing planner logic. - Keep code Python 3.11 compatible. ## Done when - All files are emitted with full contents using the delimiter. - Placeholders are present where specified. - A user can fill variables, apply manifests, build image, and run a local Wikipedia-backed deep-research workflow end to end.