hfutils.index.validate

hf_tar_item_validate

hfutils.index.validate.hf_tar_item_validate(file_item: RepoFile, size: int, hash_: str | None = None, hash_lfs: str | None = None)[source]

Validate a file item in a tar archive.

This function checks if the file item matches the expected size and hash.

Parameters:

  • file_item (RepoFile) – The file item from the Hugging Face repository.

  • size (int) – The expected size of the file.

  • hash (str, optional) – The expected SHA-1 hash of the file.

  • hash_lfs (str, optional) – The expected SHA-256 hash of the file if stored in LFS.

Returns:

True if the file item is valid, False otherwise.

Return type:

bool

hf_tar_validate

hfutils.index.validate.hf_tar_validate(repo_id: str, archive_in_repo: str, repo_type: Literal['dataset', 'model', 'space'] = 'dataset', revision: str = 'main', idx_repo_id: str | None = None, idx_file_in_repo: str | None = None, idx_repo_type: Literal['dataset', 'model', 'space'] | None = None, idx_revision: str | None = None, hf_token: str | None = None)[source]

Validate a tar archive in a Hugging Face repository.

This function validates if the tar archive in the Hugging Face repository matches the expected size and hash.

Parameters:

  • repo_id (str) – The ID of the Hugging Face repository.

  • archive_in_repo (str) – The path to the tar archive in the repository.

  • repo_type (RepoTypeTyping, optional) – The type of the Hugging Face repository, defaults to ‘dataset’.

  • revision (str, optional) – The revision of the repository, defaults to ‘main’.

  • idx_repo_id (Optional[str], optional) – The ID of the repository where the index file is stored.

  • idx_file_in_repo (Optional[str], optional) – The path to the index file in the repository.

  • idx_repo_type (Optional[RepoTypeTyping], optional) – The type of the repository where the index file is stored.

  • idx_revision (Optional[str], optional) – The revision of the repository where the index file is stored.

  • hf_token (Optional[str], optional) – The Hugging Face token for authentication, defaults to None.

Raises:

  • EntryNotFoundError – If the specified entry is not found in the repository.

  • IsADirectoryError – If the specified entry is a directory.

Returns:

True if the tar archive is valid, False otherwise.

Return type:

bool