[Mikola Lysenko (mikolalysenko)]

Problem

CI is healthy this run: the merge queue is empty, every merge_group run since 11-09 passed, main's latest push run and today's nightly are green (the nightly's first green since 10-27), and nothing is queued. So this run finishes the last open item from #1919 that #2054 left for a follow-up: python crawler unit tests that pass only because CI never activates a venv.

Reproduced on origin/main (c09cb0e), cargo test -p socket-patch-core --lib crawlers::python_crawler:

env	result
clean	102 passed
VIRTUAL_ENV=<a real venv>	5 failed: test_crawl_all_python, test_crawl_all_not_poisoned_by_body_headers, test_crawl_all_recovers_metadata_less_package, base_copy_behind_a_system_site_venv_is_crawled, venv_copy_hides_the_base_copy_of_the_same_package
CONDA_PREFIX=<a real venv>	1 failed: poetry_out_of_tree_virtualenvs_are_discovered_without_a_dot_venv

A contributor working from an activated venv or conda env gets red tests on a clean checkout.

Root cause

PythonCrawler::crawl_all and get_site_packages_paths build their env lookup from std::env::var, and an activated VIRTUAL_ENV beats the fixture's ./.venv. The Poetry test already clears VIRTUAL_ENV under #[serial], but not CONDA_PREFIX, which poetry_active_prefix treats the same way.

Fix

• Split out private get_site_packages_paths_with(options, var) and crawl_all_with(options, var). They take the env lookup as a closure, as local_site_packages_with already does. The public functions pass the process env exactly as before, so behaviour doesn't change.
• The five tests call the _with variants with a no_env lookup. I injected the env instead of an EnvGuard + #[serial] on purpose: mutating process env while other tests run in parallel is itself a flake source.
• The Poetry test's existing guard also clears CONDA_PREFIX.

No test is removed and no assertion changed. Each test asserts the same thing on the same fixtures.

Proof

cargo test -p socket-patch-core --lib crawlers:: after the change:

env	result
clean	798 passed
VIRTUAL_ENV=<venv>	798 passed
CONDA_PREFIX=<venv>	798 passed
VIRTUAL_ENV + CONDA_PREFIX + CONDA_DEFAULT_ENV=x	798 passed

• crawlers::python_crawler under VIRTUAL_ENV+CONDA_PREFIX, 20 repeats: 0 failures.
• cargo clippy --locked -p socket-patch-core --all-features -- -D warnings (CI's clippy) is clean. With --all-targets the set of lints per file is identical to main (all pre-existing, in test code).
• rustfmt --check passes on the touched file.

The public crawl_all stays covered by randomized_site_packages_match_the_async_oracle and the CLI integration suites.

Together with #2054 this meets #1919's acceptance criteria for crawlers::, apart from the two npm_crawler tests #2054 lists, which need the yarn env threaded through NpmCrawler.

Refs #1919

---

Generated by Claude Code

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

PR Summary

Low Risk
Production behavior is unchanged; only test helpers and internal _with methods were added, with no assertion changes.

Overview
Python crawler tests were failing when the developer had an activated venv or conda env because get_site_packages_paths and crawl_all always read VIRTUAL_ENV / CONDA_PREFIX from the real process environment.

This PR introduces injectable _with variants (get_site_packages_paths_with, crawl_all_with) that take an env lookup closure (matching local_site_packages_with). The public APIs still use std::env::var unchanged. Affected unit tests call the _with APIs with a no_env stub so fixtures aren’t overridden by the host. The Poetry virtualenv discovery test’s env guard also clears CONDA_PREFIX.

^{Reviewed by Cursor Bugbot for commit 144d4ab. Configure here.}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 144d4ab. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Ready for review at 144d4ab.

• CI: all checks finished with no failures (ci-ok, clippy, coverage, e2e shards all green; the rest skipped by path filters). No main-wide failures.
• Bugbot: reviewed 144d4ab and found nothing. No open review threads.
• Worth a reviewer's look: the private get_site_packages_paths_with / crawl_all_with seams (env lookup passed as a closure). The public entry points pass std::env::var exactly as before. Also the Poetry test's guard now clears CONDA_PREFIX.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief

What it does. Python crawler unit tests no longer depend on the host's activated venv or conda env. crawl_all and get_site_packages_paths now hand off to private *_with variants that take the env lookup as a closure. The public functions still pass the real process env, so production behaviour is unchanged. Five tests use an empty env, and the Poetry test's env guard also clears CONDA_PREFIX.

Risk: low. It touches one file. It is a pure refactor of two methods plus test-only changes. No assertion was changed and no test was removed.

Look here

• get_site_packages_paths_with: the public fn passes std::env::var and the body is unchanged
• crawl_all → crawl_all_with: same pattern
• no_env: the empty env used by the five tests
• Poetry test guard: CONDA_PREFIX cleared alongside VIRTUAL_ENV

Verified

• Read the full diff. The only process-env reads left on the local crawl path are in the public wrappers. The remaining std::env reads in the file are on global or Windows paths, which these tests don't hit.
• cargo test -p socket-patch-core --lib crawlers::python_crawler with VIRTUAL_ENV and CONDA_PREFIX set to a real venv: 102 passed on this head. On origin/main with VIRTUAL_ENV set: 5 failed, the same five listed in the description. So the tests fail without the change.
• CHANGELOG.md is untouched. CI: ci-ok and clippy are green on 144d4ab and no checks failed. Bugbot reviewed 144d4ab and found nothing. No open review threads.

Changes I made: none.
Open questions: none.

Auto-merge is armed, so approving sends it straight to the merge queue.

---

Generated by Claude Code