Add cross-file import tracking functionality

[cmyui]

Cross-File Import Tracking

Summary

This document describes the cross-file import tracking feature that was implemented for the unused import detector. The feature transforms the tool from single-file analysis to cross-file import tracking, following imports like Python's runtime and detecting unused imports across an entire project.

---

What Was Implemented

Core Features

1. Cross-file import tracking - Follow imports from an entry point, building an import graph
2. Re-export detection - Imports used by other files are NOT flagged as unused
3. Cascade detection - When an import becomes unused, cascade to find additional unused imports
4. Implicit re-export warnings - Warn when imports are re-exported without being in __all__
5. Circular import detection - Detect and report circular import chains
6. Unreachable file warnings - Warn about files that become truly dead code after fixing imports
7. Submodule traversal - Handle from pkg import submodule where submodule isn't in pkg/__init__.py

CLI Interface (Actual)

# Cross-file mode (default) - pass entry point file or directory
remove-unused-imports main.py                    # Entry point file
remove-unused-imports src/                       # Directory (analyzes all .py files)

# With options
remove-unused-imports main.py --fix              # Auto-fix unused imports
remove-unused-imports main.py --quiet            # Only show summary
remove-unused-imports main.py --warn-implicit-reexports  # Warn about implicit re-exports
remove-unused-imports main.py --warn-circular    # Warn about circular imports
remove-unused-imports main.py --warn-unreachable # Warn about unreachable files

# Single-file mode (original behavior, no cross-file tracking)
remove-unused-imports --single-file myfile.py
remove-unused-imports --single-file src/

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Entry Point                              │
│                      main.py or src/                             │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Module Resolver (_resolution.py)               │
│  - Resolves import statements to file paths                      │
│  - Handles relative imports (from . import x)                    │
│  - Detects external modules (stdlib/third-party)                 │
│  - Prevents resolution beyond source root                        │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Graph Builder (_graph.py)                      │
│  - BFS from entry point, following imports                       │
│  - Builds ImportGraph with nodes (files) and edges (imports)     │
│  - Handles submodule traversal for implicit imports              │
│  - Cycle detection (Tarjan's algorithm)                          │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                Cross-File Analyzer (_cross_file.py)              │
│  - Runs single-file analysis on each module                      │
│  - Detects re-export-only imports (in __all__ but unused locally)│
│  - Detects implicit re-export-only imports (__init__.py)         │
│  - CASCADE: Iterates until stable, tracking file reachability    │
│  - Finds implicit re-exports (not in __all__)                    │
│  - Aggregates external module usage                              │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Output Formatter (_format.py)                  │
│  - Groups unused imports by file and line                        │
│  - Formats implicit re-export warnings                           │
│  - Formats circular import warnings                              │
│  - Formats unreachable file warnings                             │
│  - Summary with totals                                           │
└─────────────────────────────────────────────────────────────────┘

---

Data Structures

_data.py

@dataclass
class ImportInfo:
    name: str              # Bound name (considering aliases)
    original_name: str     # Original name before aliasing
    module: str            # Module being imported from
    lineno: int
    is_from_import: bool
    level: int             # 0=absolute, 1=from ., 2=from .., etc.

@dataclass
class ModuleInfo:
    file_path: Path
    module_name: str          # "mypackage.submodule.utils"
    is_package: bool          # True for __init__.py
    imports: list[ImportInfo]
    exports: set[str]         # Names in __all__
    defined_names: set[str]   # Classes, functions, variables defined

@dataclass
class ImportEdge:
    importer: Path
    imported: Path | None     # None = external module
    module_name: str          # The module string from import statement
    names: set[str]           # Names being imported
    is_external: bool
    level: int

@dataclass
class ImplicitReexport:
    source_file: Path
    import_name: str
    used_by: set[Path]

_cross_file.py

@dataclass
class CrossFileResult:
    unused_imports: dict[Path, list[ImportInfo]]
    implicit_reexports: list[ImplicitReexport]
    external_usage: dict[str, set[Path]]  # module -> files using it
    circular_imports: list[list[Path]]
    unreachable_files: set[Path]          # Truly dead code files

---

Key Algorithms

Cascade Detection

The cascade algorithm iteratively finds unused imports until stable:

1. Run single-file analysis on each module
2. Find "reexport-only" imports (in __all__ but unused locally)
3. Find "implicit-reexport-only" imports (__init__.py without __all__)
4. LOOP until no changes:
   a. Update file reachability based on removed imports
   b. Find re-exported imports (excluding unreachable files as consumers)
   c. Mark imports as removed if:
      - Unused locally AND not re-exported, OR
      - Reexport-only AND not re-exported, OR
      - Implicit-reexport-only AND not re-exported
5. Build final unused_imports from stable removed set

Unreachable File Detection

Two concepts are tracked separately:

Concept	Used For	Criteria
Potentially unreachable	Cascade detection	No direct import edges after removing unused imports
Truly unreachable	User warning	No direct edges AND no reachable ancestor packages

This distinction handles patterns where import pkg + pkg.submodule.x keeps submodules accessible at runtime even if from pkg import submodule is unused.

Submodule Traversal

When processing from pkg import name:

1. Resolve pkg to pkg/__init__.py
2. Check if name is already in pkg/__init__.py (imports or definitions)
3. If NOT, try resolving pkg.name as a submodule
4. If found, traverse it (handles implicit submodule imports)

---

Files Created/Modified

File	Action	Description
_data.py	Modified	Added ModuleInfo, ImportEdge, ImplicitReexport, level field
_resolution.py	Created	Module resolver, external module detection
_graph.py	Created	Import graph, graph builder, cycle detection, submodule traversal
_cross_file.py	Created	Cross-file analyzer with cascade detection
_format.py	Created	Output formatting for CLI
_ast_helpers.py	Modified	Added level to ImportExtractor, added collect_dunder_all_names
_main.py	Modified	New CLI interface, cross-file mode as default
_detection.py	Modified	Added ignore_all parameter for reexport-only detection
tests/cross_file_test.py	Created	22 tests for cross-file functionality

---

Test Coverage

Cross-file tests (tests/cross_file_test.py)

• Re-export detection: test_reexported_import_not_unused, test_chain_of_reexports, test_partial_reexport
• Implicit re-exports: test_implicit_reexport_detected
• External usage: test_external_usage_aggregated
• Explicit all: test_explicit_reexport_not_flagged
• Circular imports: test_circular_import_detected, test_no_circular_when_none
• Defined names: test_defined_name_not_reexport
• Cascade detection:
  • test_cascade_unused_when_consumer_removed
  • test_cascade_partial_chain
  • test_cascade_multiple_consumers
  • test_cascade_reexport_only_via_dunder_all
  • test_cascade_reexport_only_keeps_used
  • test_cascade_file_unreachable
  • test_cascade_file_still_reachable_via_other_path
• Submodule traversal: test_submodule_not_in_init_is_traversed
• Unreachable files:
  • test_submodule_not_unreachable_when_parent_reachable
  • test_truly_unreachable_file_is_reported
  • test_cascade_works_for_potentially_unreachable_but_not_reported
  • test_cascade_detects_more_unused_via_potentially_unreachable

Total: 333 tests passing

---

Example Output

$ remove-unused-imports app.py --warn-unreachable

app.py
   15: Unused 'Dict' from 'typing'

utils/__init__.py
     7: Unused 'EmailClient' from 'utils.email'
     8: Unused 'CacheManager' from 'utils.cache'

models/__init__.py
    42: Unused 'LegacyModel' from 'models.legacy'

───────────────────────────────────────────────────────────────────────────────
Unreachable Files (will become dead code after fixing imports)
───────────────────────────────────────────────────────────────────────────────
  • orphaned_module.py

═══════════════════════════════════════════════════════════════════════════════
Found 45 unused import(s) in 12 file(s), 1 unreachable file(s)
═══════════════════════════════════════════════════════════════════════════════

---

Edge Cases Handled

1. Circular imports: Detected via Tarjan's algorithm, reported with --warn-circular
2. TYPE_CHECKING blocks: Treated as annotation-only imports (existing behavior)
3. __init__.py re-exports: Tracked via exports field (names in __all__)
4. Implicit submodule imports: from pkg import submod where submod isn't in __init__.py
5. Case-insensitive filesystems: Using Path.samefile() for comparisons (macOS)
6. Parent package imports: import pkg + pkg.submod.x keeps submodules accessible

---

Known Limitations

1. Dynamic imports: importlib.import_module() cannot be analyzed statically
2. Star imports: from x import * uses __all__ if available, otherwise skipped
3. Namespace packages: PEP 420 namespace packages not fully supported
4. Conditional imports: Imports inside if blocks are always considered
5. Runtime attribute access: pkg.submod access without explicit import is not tracked

---

Performance

Tested on a real-world web application (~100 Python files):

• Graph construction: < 1 second
• Cross-file analysis with cascade: < 2 seconds
• Detects 250+ unused imports in one shot

---

Future Improvements

1. Parallel graph construction: Process files concurrently
2. Incremental analysis: Cache graph between runs
3. IDE integration: LSP server for real-time feedback
4. Attribute access tracking: Detect pkg.submod usage patterns

[Copilot]

Pull request overview

This PR introduces cross-file import tracking functionality, transforming the tool from single-file analysis to understanding imports across an entire codebase. The implementation adds the ability to detect re-exported imports (where a module imports something and other files import it from that module), identify implicit re-exports (re-exports not declared in __all__), and detect circular import chains.

Changes:

• Implements a complete module resolution system that mirrors Python's import mechanism
• Adds import graph construction to track dependencies between files
• Introduces cross-file analysis that identifies re-exported imports to avoid false positives
• Updates CLI to default to cross-file mode with --single-file flag for backward compatibility

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
CROSS_FILE_PLAN.md	Comprehensive design document outlining architecture, data structures, and implementation phases
remove_unused_imports/_data.py	Adds ModuleInfo, ImportEdge, and ImplicitReexport data classes; extends ImportInfo with level field
remove_unused_imports/_resolution.py	New module implementing Python-like import resolution with support for relative imports and PYTHONPATH
remove_unused_imports/_graph.py	New module for building and analyzing import graphs with cycle detection and topological ordering
remove_unused_imports/_cross_file.py	Implements cross-file analysis to identify re-exports and aggregate usage across modules
remove_unused_imports/_main.py	Updates CLI to support cross-file mode by default with new flags for warnings and backward compatibility
remove_unused_imports/_ast_helpers.py	Adds level parameter to ImportExtractor for relative import support
remove_unused_imports/init.py	Exports new public APIs for cross-file analysis
tests/resolution_test.py	Comprehensive tests for module resolution including relative imports and PYTHONPATH
tests/graph_test.py	Tests for import graph construction, cycle detection, and topological ordering
tests/cross_file_test.py	Tests for re-export detection, implicit re-exports, and chained re-exports
tests/cli_test.py	Tests for CLI with cross-file mode flags and backward compatibility
.pre-commit-config.yaml	Increases max line length from 100 to 110 characters

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.