easyidp.data Advanced API¶

This page documents dataset internals and downloader helpers for contributors and advanced integrations. Ordinary users should prefer Lotus, ForestBirds, TestData, and list_datasets on the main Data page.

Downloader Extension Points¶

easyidp.data.downloader.download_dataset(dataset, mirror='auto', force=False, progress=True)¶

Download and extract a dataset.

Parameters:

dataset (Dataset) – Dataset instance with mirrors and archive metadata.
mirror (str, optional) – Mirror name or "auto" to pick the first available mirror.
force (bool, optional) – Re-download even when the dataset is already ready.
progress (bool, optional) – Show a progress bar during download.

Returns:

JSON-friendly result with keys name, root, archive, downloaded, extracted, ready.

Return type:

dict

easyidp.data.downloader.safe_extract_zip(archive, dest)¶

Extract a zip archive, rejecting path-traversal members.

Parameters:

archive (Path) – Path to the zip file.
dest (Path) – Directory to extract into.

Raises:

RuntimeError – If any member resolves outside dest.

Dataset Internals¶

class easyidp.data.dataset.Dataset(manifest_name, cache_root=None, notify_missing=True)¶

EasyIDP dataset backed by a JSON manifest.

Parameters:

manifest_name (str) – Name of the JSON manifest without extension (e.g. "lotus").
cache_root (Path or str, optional) – Root directory for cached datasets. Defaults to the value returned by easyidp.config.get("data_dir")().
notify_missing (bool, optional) – Retained for backward compatibility only; no longer logs warnings.

Examples

>>> dataset = Dataset("lotus")
>>> dataset.name
'lotus'
>>> dataset.path("shp").name
'plots.shp'

download(mirror='auto', force=False, progress=True)¶

Download this dataset to cache_root.

Parameters:

mirror (str, optional) – Mirror name or "auto" (default) to pick the first available.
force (bool, optional) – Re-download even if the dataset is ready.
progress (bool, optional) – Show a progress bar during download.

dry_run()¶

Return a JSON-friendly summary dict without touching the network.

Returns:: Summary with keys name, description, root, ready, needs_download, size_bytes, missing. All path values are plain strings.
Return type:: dict

is_ready()¶

Return True when every ready_check file exists on disk.

If no ready_check list is present, all files are checked.

property name¶: Dataset manifest name.

path(key)¶

Return the absolute Path for a dotted file key.

Parameters:: key (str) – Dotted file key (e.g. "metashape.project").
Returns:: Absolute path.
Return type:: Path

property root¶: Extracted dataset directory.

class easyidp.data.dataset._PathNamespace(root, tree)¶

Recursively expose nested file mappings as path attributes.

String leaf values become root / value. Intermediate mappings become nested _PathNamespace objects sharing the same root.

Parameters:

root (Path) – Absolute base directory.
tree (dict) – Nested dict of relative path strings.

Examples

>>> from pathlib import Path
>>> ns = _PathNamespace(Path("/data"), {"ms": {"dom": "outputs/dom.tif"}})
>>> ns.ms.dom
PosixPath('/data/outputs/dom.tif')

easyidp.data.dataset._insert_path(obj, files, root)¶

Build _PathNamespace attributes from dotted file keys on obj.

Parameters:

obj (object) – Target object (typically a Dataset instance).
files (Mapping) – Flat dotted-key → relative-path mapping.
root (Path) – Absolute base directory.

Examples

>>> class Paths:
...     pass
>>> from pathlib import Path
>>> obj = Paths()
>>> _insert_path(obj, {"pix4d.dom": "outputs/dom.tif"}, Path("/data"))
>>> obj.pix4d.dom
PosixPath('/data/outputs/dom.tif')

easyidp.data.dataset._load_manifest(path)¶

Load and parse a JSON manifest file.

Parameters:

path (Path) – Path to the JSON manifest.

Returns:

Parsed manifest data.

Return type:

dict

Raises:

FileNotFoundError – If the manifest file does not exist.
ValueError – If the JSON is malformed.

Examples

>>> manifest = _load_manifest(_MANIFEST_DIR / "lotus.json")
>>> manifest["spec"]["name"]
'lotus'

easyidp.data.dataset._validate_attr_name(name)¶

Check that a dotted file key does not conflict with reserved Dataset attributes.

Parameters:: name (str) – Dotted key (e.g. "metashape.project").
Raises:: ValueError – If any segment of name is a reserved attribute.

Examples

>>> _validate_attr_name("metashape.project")
>>> _validate_attr_name("root")
Traceback (most recent call last):
...
ValueError: file key 'root' contains reserved attribute 'root'

easyidp.data.dataset._validate_manifest(data)¶

Validate manifest structure and raise ValueError on problems.

Parameters:: data (dict) – Loaded manifest dictionary.
Raises:: ValueError – If required fields are missing or have wrong types.

Examples

>>> manifest = _load_manifest(_MANIFEST_DIR / "lotus.json")
>>> _validate_manifest(manifest)

Downloader Internals¶

easyidp.data.downloader._download_gdrive(file_id, archive, progress)¶

Download a file from Google Drive via gdown.

Parameters:

file_id (str) – Google Drive file ID.
archive (Path) – Target file path for the downloaded archive.
progress (bool) – Show a progress bar.

Raises:

RuntimeError – If gdown is not installed (with install hint).

easyidp.data.downloader._download_modelscope(mirror_config, archive, progress)¶

Download a dataset archive from ModelScope.

Parameters:

mirror_config (Mapping) – Mirror config with dataset_repo and file_path keys.
archive (Path) – Target file path for the downloaded archive.
progress (bool) – Print EasyIDP-level source and target paths. ModelScope controls its own progress output internally.

Examples

>>> cfg = {"dataset_repo": "owner/repo", "file_path": "archive.zip"}
>>> _download_modelscope(cfg, Path("archive.zip"), progress=False)

Raises:: RuntimeError – If ModelScope is missing or the downloaded file fails verification.

easyidp.data.downloader._fetch_modelscope_file_info(mirror_config)¶

Fetch ModelScope file size and SHA256 metadata.

Parameters:: mirror_config (Mapping) – Mirror config with dataset_repo and file_path keys.
Returns:: Download verification metadata with size_bytes and sha256.
Return type:: dict
Raises:: RuntimeError – If the target file is absent from ModelScope metadata.

Examples

>>> cfg = {"dataset_repo": "owner/repo", "file_path": "archive.zip"}
>>> _fetch_modelscope_file_info(cfg)
{'size_bytes': 1024, 'sha256': '...'}

easyidp.data.downloader._file_sha256(path)¶

Return the SHA256 hex digest for a file.

Parameters:: path (Path) – File to hash.
Returns:: Lowercase SHA256 hex digest.
Return type:: str

Examples

>>> _file_sha256(Path("archive.zip"))
'...'

easyidp.data.downloader._result(dataset, *, downloaded, extracted, ready)¶

Build a JSON-friendly download result dict.

Parameters:

dataset (Dataset) – Dataset instance.
downloaded (bool) – Whether the archive was freshly downloaded.
extracted (bool) – Whether the archive was freshly extracted.
ready (bool) – Whether all required files are now present.

Returns:

Result with name, root, archive, downloaded, extracted, ready as JSON-friendly values.

Return type:

dict

easyidp.data.downloader._select_mirror(mirrors, mirror)¶

Select a mirror key.

Parameters:

mirrors (Mapping) – Available mirrors dict from the manifest.
mirror (str) – Requested mirror name or "auto".

Returns:

Selected mirror key.

Return type:

str

Raises:

ValueError – If mirror is "auto" and no mirrors exist, or if the named mirror is not found.

easyidp.data.downloader._verify_downloaded_file(path, mirror_config)¶

Verify downloaded archive size and SHA256 metadata.

Parameters:

path (Path) – Downloaded file path.
mirror_config (Mapping) – Mirror config with optional size_bytes and sha256 values.

Raises:

RuntimeError – If the downloaded file size or SHA256 does not match the manifest.

Examples

>>> _verify_downloaded_file(Path("archive.zip"), {})