core_lens.utils.paths#

URI-agnostic path helpers built on pyarrow.fs.

Abstracts over local filesystem and cloud object stores (S3, GCS, Azure) so the rest of the codebase never has to branch on URI scheme.

Cloud path notes#

For cloud URIs (s3://, gs://, abfs://, etc.) path_exists() performs a real remote metadata call (e.g. HeadObject on S3). To avoid the per-call latency when registering many entities, existence checks are skipped at _resolve() time for cloud roots and are only performed once inside _validate_entity() via the schema-read path (Polars raises a meaningful error if the file is absent). Local paths continue to receive eager existence checks as before.

Performance notes#

resolve_fs_and_path() is called on every Parquet scan, index build, and schema read. The hot path for cloud URIs previously paid pyarrow.fs.FileSystem.from_uri C++ object-construction cost on every invocation. This is now eliminated by a two-level cache:

Per-URI cache — _resolve_fs_and_path_cached() is wrapped with functools.lru_cache(). Identical URIs (same string) return the cached (FileSystem, path) tuple in ~50 ns instead of ~50–200 µs.
LocalFileSystem singleton — pafs.LocalFileSystem() is allocated once at module import time; local-path calls reuse the same object.

Cache diagnostics:

from core_lens.utils.paths import _resolve_fs_and_path_cached
_resolve_fs_and_path_cached.cache_info()   # hits / misses / maxsize

Cache invalidation (tests / credential rotation):

_resolve_fs_and_path_cached.cache_clear()

Functions#

`is_cloud_uri`(→ bool)	Return `True` if uri refers to a cloud object store.
`resolve_fs_and_path`(→ tuple[pyarrow.fs.FileSystem, str])	Resolve a URI to a `(FileSystem, path)` pair.
`path_exists`(→ bool)	Return `True` if the path or object exists on the filesystem.
`join_uri`(→ str)	Join rel onto root, handling both local paths and cloud URIs.

Module Contents#

core_lens.utils.paths.is_cloud_uri(uri: str) → bool#

Return True if uri refers to a cloud object store.

Parameters:: uri (str) – Any path or URI string.
Returns:: True for URIs starting with a recognised cloud scheme.
Return type:: bool

core_lens.utils.paths.resolve_fs_and_path(uri: str) → tuple[pyarrow.fs.FileSystem, str]#

Resolve a URI to a (FileSystem, path) pair.

Delegates to pyarrow.fs.FileSystem.from_uri() for cloud URIs and returns a pyarrow.fs.LocalFileSystem for plain local paths.

Results are cached by _resolve_fs_and_path_cached() so repeated calls with the same URI are effectively free (~50 ns per call after the first hit).

Parameters:: uri (str) – A local path or cloud URI (e.g. s3://bucket/prefix).
Returns:: The resolved filesystem and the normalised path within that filesystem.
Return type:: tuple[pyarrow.fs.FileSystem, str]

core_lens.utils.paths.path_exists(uri: str) → bool#

Return True if the path or object exists on the filesystem.

Uses pyarrow.fs.FileSystem.get_file_info() so it works for both local paths and cloud URIs.

Parameters:: uri (str) – A local path or cloud URI.
Returns:: True if the path exists (file or directory).
Return type:: bool

core_lens.utils.paths.join_uri(root: str, rel: str) → str#

Join rel onto root, handling both local paths and cloud URIs.

For cloud roots the join is a simple string concatenation with / as separator (object-store “directories” are just key prefixes). For local roots pathlib.Path.__truediv__() is used for correct OS handling.

Parameters:

root (str) – The root directory path or cloud URI prefix.
rel (str) – A relative sub-path to append.

Returns:

The joined URI or absolute path string.

Return type:

str