thoth.storages.graph package

Submodules

thoth.storages.graph.dgraph module

A Dgraph server adapter communicating via gRPC.

class thoth.storages.graph.dgraph.GraphDatabase(hosts: List[str] = None, tls_path: str = None)[source]

Bases: thoth.storages.base.StorageBase

A dgraph server adapter communicating via gRPC.

DEFAULT_HOST = 'localhost:9080'
ENVVAR_HOST_NAME = 'GRAPH_SERVICE_HOST'
TLS_PATH = None
adviser_document_id_exist(adviser_document_id: str) → bool[source]

Check if there is a adviser document record with the given id.

analysis_document_id_exist(analysis_document_id: str) → bool[source]

Check if there is an analysis document record with the given id.

analysis_records_exist(analysis_document: dict) → bool[source]

Check whether the given analysis document records exist in the graph database.

build_software_environment_analyses_listing(build_software_environment_name: str, start_offset: int = 0, count: int = 100, convert_datetime: bool = True) → list[source]

Get listing of analyses available for the given software environment for build.

build_software_environment_listing(start_offset: int = 0, count: int = 100) → list[source]

Get listing of software environments available for build.

property client

Retrieve client for communicating with DGraph instance.

compute_python_package_version_avg_performance(packages: Set[tuple], *, run_software_environment: dict = None, hardware_specs: dict = None) → float[source]

Get average performance of Python packages on the given runtime environment.

We derive this average performance based on software stacks we have evaluated on the given software environment for run including the given package in specified version. There are also included stacks that failed for some reason that have negative performance impact on the overall value.

There are considered software stacks that include packages listed, they can however include also other packages.

Optional parameters additionally slice results - e.g. if run_software_environment is set, it picks only results that match the given parameters criteria.

connect()[source]

Connect to a Dgraph via gRPC.

classmethod create(host: str)[source]

Create a graph adapter, only for one host (syntax sugar).

create_advised_software_stack_pipfile(adviser_document_id: str, pipfile_locked: dict, *, advised_stack_index: int, performance_score: float, overall_score: float, run_software_environment: thoth.storages.graph.models.UserRunSoftwareEnvironment) → thoth.storages.graph.models.AdvisedSoftwareStack[source]

Create an advised software stack entry from Pipfile.lock.

create_inspection_software_stack_pipfile(document_id: str, pipfile_locked: dict) → thoth.storages.graph.models.InspectionSoftwareStack[source]

Create an inspection software stack entry from Pipfile.lock.

create_python_cve_record(package_name: str, package_version: str, index_url: str, *, record_id: str, version_range: str, advisory: str, cve: str = None) → Tuple[thoth.storages.graph.models.CVE, bool][source]

Store information about a CVE in the graph database for the given Python package.

create_python_package_requirement(requirements: dict) → List[thoth.storages.graph.models.PythonPackageRequirement][source]

Create requirements for un-pinned Python packages.

create_python_package_version_entity(package_name: str, package_version: str, index_url: str, *, only_if_package_seen: bool = False) → Optional[Tuple[thoth.storages.graph.models.PythonPackageVersionEntity, bool]][source]

Create a Python package version entity in the graph database.

create_python_packages_pipfile(pipfile_locked: dict, run_software_environment: thoth.storages.graph.models.UserRunSoftwareEnvironment = None) → List[thoth.storages.graph.models.PythonPackageVersion][source]

Create Python packages from Pipfile.lock entries and return them.

create_user_software_stack_pipfile(adviser_document_id: str, pipfile_locked: dict, run_software_environment: thoth.storages.graph.models.UserRunSoftwareEnvironment = None) → thoth.storages.graph.models.UserSoftwareStack[source]

Create a user software stack entry from Pipfile.lock.

dependency_monkey_document_id_exist(dependency_monkey_document_id: str) → bool[source]

Check if the given dependency monkey report record exists in the graph database.

disconnect()[source]

Close all connections - disconnect from remote.

drop_all() → None[source]

Drop all data present inside Dgraph instance.

get_all_pi_per_framework_count(framework: str) → dict[source]

Retrieve dictionary with number of Performance Indicators per ML Framework in the graph database.

get_all_python_package_version_hashes_sha256(package_name: str, package_version: str) → list[source]

Get hashes for a Python package per index.

get_all_python_packages_count(without_error: bool = True) → int[source]

Retrieve number of Python packages stored in the graph database.

get_all_versions_python_package(package_name: str, index_url: str = None, *, only_known_index: bool = False, os_name: str = None, os_version: str = None, python_version: str = None, without_error: bool = True, only_solved: bool = False) → List[Tuple[str, str]][source]

Get all versions available for a Python package.

get_analysis_metadata(analysis_document_id: str) → dict[source]

Get metadata stored for the given analysis document.

get_analyzer_documents_count() → int[source]

Get number of image analysis documents synced into graph.

get_error_python_packages_count(*, unsolvable: bool = False, unparseable: bool = False) → int[source]

Retrieve number of Python packages stored in the graph database with error flag.

get_number_of_each_vertex_in_graph() → dict[source]

Retrieve dictionary with number of vertices per vertex label in the graph database.

get_python_cve_records(package_name: str, package_version: str) → List[dict][source]

Get known vulnerabilities for the given package-version.

get_python_package_index_urls() → set[source]

Retrieve all the URLs of registered Python package indexes.

async get_python_package_tuple(python_package_node_id: int) → Dict[int, tuple][source]

Get Python’s package name, package version, package index tuple for the given package id.

get_python_package_tuples(python_package_node_ids: Set[int]) → Dict[int, Tuple[str, str, str]][source]

Get package name, package version and index URL for each python package node.

This query is good to be used in conjunction with query retrieving transitive dependencies. The main benefit of this function is that it performs all the queries in an event loop per each package.

get_python_package_version_hashes_sha256(package_name: str, package_version: str, index_url: str) → List[str][source]

Get hashes for a Python package in specified version.

get_python_packages() → Set[str][source]

Retrieve listing of all Python packages known to graph database instance.

get_python_packages_for_index(index_url: str) → Set[str][source]

Retrieve listing of Python packages known to graph database instance for the given index.

get_solver_documents_count() → int[source]

Get number of solver documents synced into graph.

has_python_solver_error(package_name: str, package_version: str, index_url: str, *, os_name: str, os_version: str, python_version: str) → bool[source]

Retrieve information whether the given package has any solver error.

property hosts

Get hosts configured for this adapter.

initialize_schema() → None[source]

Initialize Dgraph’s schema.

inspection_document_id_exist(inspection_document_id: str) → bool[source]

Check if there is an inspection document record with the given id.

is_connected() → bool[source]

Check if we are connected to a remote Dgraph instance.

static normalize_python_package_name(package_name: str) → str[source]

Normalize Python package name based on PEP-0503.

package_analysis_document_id_exist(package_analysis_document_id: str) → bool[source]

Check if there is a package analysis document record with the given id.

static parse_python_solver_name(solver_name: str) → dict[source]

Parse os and Python identifiers encoded into solver name.

provenance_checker_document_id_exist(provenance_checker_document_id: str) → bool[source]

Check if there is a provenance-checker document record with the given id.

python_package_exists(package_name: str) → bool[source]

Check if the given Python package exists regardless of version.

python_package_index_listing() → list[source]

Get listing of Python package indexes registered in the graph database database.

python_package_version_exists(package_name: str, package_version: str, index_url: str = None, solver_name: str = None) → bool[source]

Check if the given Python package version exists in the graph database.

If optional solver_name parameter is set, the call answers if the given package was solved by the given solver. Otherwise, any solver run is taken into account.

register_python_package_index(url: str, warehouse_api_url: str = None, verify_ssl: bool = True) → bool[source]

Register the given Python package index in the graph database.

retrieve_dependent_packages(package_name: str) → dict[source]

Get mapping package name to package version of packages that depend on the given package.

retrieve_document_list_of_unsolvable_python_packages() → list[source]

Retrieve a dictionary mapping package names to versions of packages that were marked as unsolvable.

retrieve_solved_python_packages(count: int = 10, start_offset: int = 0, solver_name: str = None) → dict[source]

Retrieve a dictionary mapping package names to versions for dependencies that were already solved.

Using count and start_offset is possible to change pagination. Using solver_name argument the query narrows down to packages that were resolved by the given solver.

retrieve_transitive_dependencies_python(package_name: str, package_version: str, index_url: str, *, os_name: str = None, os_version: str = None, python_version: str = None, _ids_map: dict = None, _dependencies_map: dict = None) → list[source]

Get all transitive dependencies for the given package by traversing dependency graph.

It’s much faster to retrieve just dependency ids for the transitive dependencies as most of the time is otherwise spent in serialization and deserialization of query results. The ids are obtained later on (kept in ids map, see bellow).

The ids map represents a map to optimize number of retrievals - not to perform duplicate queries into graph instance.

retrieve_transitive_dependencies_python_multi(*package_tuples, os_name: str = None, os_version: str = None, python_version: str = None) → dict[source]

Get all transitive dependencies for a given set of packages by traversing the dependency graph.

retrieve_unanalyzed_python_package_versions(start_offset: int = 0, count: int = 100) → List[dict][source]

Retrieve a list of package names, versions and index urls that were not analyzed yet by package-analyzer.

retrieve_unparseable_python_packages() → dict[source]

Retrieve a dictionary mapping package names to versions of packages that couldn’t be parsed by solver.

retrieve_unsolvable_python_packages(solver_name: str = None) → dict[source]

Retrieve a dictionary mapping package names to versions of packages that were marked as unsolvable.

retrieve_unsolvable_python_packages_per_run_software_environment(solver_name: str) → dict[source]

Retrieve a dictionary mapping package names to versions of packages that were marked as unsolvable.

The result is given for a specific run software environment (OS + python version)

retrieve_unsolved_python_packages(solver_name: str) → dict[source]

Retrieve a dictionary mapping package names to versions that dependencies were not yet resolved.

Using solver_name argument the query narrows down to packages that were not resolved by the given solver.

retrieve_unsolved_python_packages_count(solver_name: str) → int[source]

Retrieve number of unsolved Python packages for the given solver.

run_software_environment_analyses_listing(run_software_environment_name: str, start_offset: int = 0, count: int = 100, convert_datetime: bool = True, is_user_run: bool = False) → list[source]

Get listing of analyses available for the given software environment for run.

run_software_environment_listing(start_offset: int = 0, count: int = 100, is_user_run: bool = False) → list[source]

Get listing of software environments available for run.

solver_document_id_exist(solver_document_id: str) → bool[source]

Check if there is a solver document record with the given id.

solver_records_exist(solver_document: dict) → bool[source]

Check if the given solver document record exists.

sync_adviser_result(document: dict) → None[source]

Sync adviser result into graph database.

sync_analysis_result(document: dict) → None[source]

Sync the given analysis result to the graph database.

sync_dependency_monkey_result(document: dict) → None[source]

Sync reports of dependency monkey runs.

sync_inspection_result(document) → None[source]

Sync the given inspection document into the graph database.

sync_package_analysis_result(document: dict) → None[source]

Sync the given package analysis result to the graph database.

sync_provenance_checker_result(document: dict) → None[source]

Sync provenance checker results into graph database.

sync_solver_result(document: dict) → None[source]

Sync the given solver result to the graph database.

thoth.storages.graph.models module

Graph database schema.

This module contains just “core” parts of the graph database schema. Other modules present in this package capture standalone parts - see for example performance indicators.

class thoth.storages.graph.models.Advised(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

A relation stating advised software stack in an adviser run.

class thoth.storages.graph.models.AdvisedSoftwareStack(uid: int = None, *, adviser_document_id: str = None, advised_stack_index: int = None, performance_score: float = None, overall_score: float = None)[source]

Bases: thoth.storages.graph.models.SoftwareStackBase

A software stack as produced by adviser (the output of recommendation engine).

advised_stack_index
adviser_document_id
overall_score
performance_score
class thoth.storages.graph.models.AdviserRun(uid: int = None, *, adviser_document_id: str = None, adviser_datetime: datetime.datetime = None, adviser_version: str = None, adviser_name: str = None, count: int = None, limit: int = None, origin: str = None, debug: bool = None, limit_latest_versions: int = None, adviser_error: bool = None, recommendation_type: str = None, requirements_format: str = None, duration: int = None, advised_configuration_changes: bool = None, additional_stack_info: bool = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single adviser run.

additional_stack_info
advised_configuration_changes
adviser_datetime
adviser_document_id
adviser_error
adviser_name
adviser_version
count
debug
duration
limit
limit_latest_versions
origin
recommendation_type
requirements_format
class thoth.storages.graph.models.AdviserRunSoftwareEnvironmentInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

A relation capturing software environment for run used in advises.

class thoth.storages.graph.models.AdviserStackInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

A relation stating user’s software stack coming in to the recommendation engine.

class thoth.storages.graph.models.AnalyzedBy(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

An edge representing an image analysis.

class thoth.storages.graph.models.BuildSoftwareEnvironment(uid: int = None, *, environment_name: str = None, python_version: str = None, image_name: str = None, image_sha: str = None, os_name: str = None, os_version: str = None)[source]

Bases: thoth.storages.graph.models.EnvironmentBase

Software Environment for build - container image which consists of various packages used during build.

class thoth.storages.graph.models.CVE(uid: int = None, *, advisory: str = None, cve_name: str = '-', cve_id: str = None, version_range: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

Information about a CVE.

ELEMENT_NAME = 'cve'
advisory
cve_id
cve_name
version_range
class thoth.storages.graph.models.CreatesStack(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given set of packages create a stack.

class thoth.storages.graph.models.DebDependency(uid: int = None, *, ecosystem: str = None, package_name: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A Debian dependency.

ecosystem
package_name
class thoth.storages.graph.models.DebDepends(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None, version_range: str = '*')[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Depending edge of a deb package.

version_range
class thoth.storages.graph.models.DebPackageVersion(uid: int = None, *, ecosystem: str = None, package_name: str = None, package_version: str = None, epoch: str = None, arch: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

Debian-specific package version.

arch
ecosystem
epoch
package_name
package_version
class thoth.storages.graph.models.DebPreDepends(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None, version_range: str = '*')[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Pre-depending edge of a deb package.

version_range
class thoth.storages.graph.models.DebReplaces(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None, version_range: str = '*')[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

An edge of a deb package capturing package replacement..

version_range
class thoth.storages.graph.models.DependencyMonkeyRun(uid: int = None, *, dependency_monkey_document_id: str = None, dependency_monkey_datetime: datetime.datetime = None, dependency_monkey_name: str = None, dependency_monkey_version: str = None, seed: int = None, decision: str = None, count: int = None, limit_latest_versions: bool = None, debug: bool = None, dependency_monkey_error: bool = None, duration: int = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single dependency-monkey run.

count
debug
decision
dependency_monkey_datetime
dependency_monkey_document_id
dependency_monkey_error
dependency_monkey_name
dependency_monkey_version
duration
limit_latest_versions
seed
class thoth.storages.graph.models.DependencyMonkeyRunSoftwareEnvironmentInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

A class representing software environment for run used during stack resolution on a dependency monkey run.

class thoth.storages.graph.models.DependsOn(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None, version_range: str = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Dependency between packages modeling based on ecosystem specification.

version_range
class thoth.storages.graph.models.EcosystemSolverRun(uid: int = None, *, ecosystem: str = None, solver_document_id: str = None, solver_datetime: datetime.datetime = None, solver_name: str = None, solver_version: str = None, os_name: str = None, os_version: str = None, python_version: str = None, duration: int = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single solver run.

duration
ecosystem
os_name
os_version
python_version
solver_datetime
solver_document_id
solver_name
solver_version
class thoth.storages.graph.models.EnvironmentBase(uid: int = None, *, environment_name: str = None, python_version: str = None, image_name: str = None, image_sha: str = None, os_name: str = None, os_version: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A base class for environment types.

environment_name
image_name
image_sha
os_name
os_version
python_version
class thoth.storages.graph.models.FoundFile(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None, file_path: str = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

An edge representing a filepath to the python file found in an image.

file_path
class thoth.storages.graph.models.HardwareInformation(uid: int = None, *, cpu_vendor: str = None, cpu_model: int = None, cpu_cores: int = None, cpu_model_name: str = None, cpu_family: int = None, cpu_physical_cpus: int = None, gpu_model_name: str = None, gpu_vendor: str = None, gpu_cores: int = None, gpu_memory_size: float = None, ram_size: float = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

Hardware specification and properties.

cpu_cores
cpu_family
cpu_model
cpu_model_name
cpu_physical_cpus
cpu_vendor
gpu_cores
gpu_memory_size
gpu_model_name
gpu_vendor
ram_size
class thoth.storages.graph.models.HasArtifact(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given package has the given artifact.

class thoth.storages.graph.models.HasVulnerability(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given package version has a vulnerability.

class thoth.storages.graph.models.Identified(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

An edge representing a package identified by a package-extract run.

class thoth.storages.graph.models.IncludedFile(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None, package_analysis_document_id: str = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

An edge representing file found in the given artifact.

package_analysis_document_id
class thoth.storages.graph.models.InspectionBuildSoftwareEnvironmentInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Software environment for build in inspections.

class thoth.storages.graph.models.InspectionRun(uid: int = None, *, inspection_document_id: str = None, inspection_datetime: datetime.datetime = None, amun_version: str = None, build_requests_cpu: float = None, build_requests_memory: float = None, run_requests_cpu: float = None, run_requests_memory: float = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single inspection run.

amun_version
build_requests_cpu
build_requests_memory
inspection_datetime
inspection_document_id
run_requests_cpu
run_requests_memory
class thoth.storages.graph.models.InspectionRunSoftwareEnvironmentInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Software environment for run in inspections.

class thoth.storages.graph.models.InspectionSoftwareStack(uid: int = None, *, inspection_document_id: str = None)[source]

Bases: thoth.storages.graph.models.SoftwareStackBase

A software stack which was used on Amun during inspection runs (e.g. as produced by dependency-monkey).

inspection_document_id
class thoth.storages.graph.models.InspectionStackInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

An input stack for an inspection run.

class thoth.storages.graph.models.InstalledFrom(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given PythonPackageVersion installed from the given Python entity.

class thoth.storages.graph.models.Investigated(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given artifact is investigated by the package analyzer.

class thoth.storages.graph.models.PackageAnalyzerInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

An input python-package-version entity for a package analyzer run.

class thoth.storages.graph.models.PackageAnalyzerRun(uid: int = None, *, package_analysis_document_id: str = None, package_analysis_datetime: datetime.datetime = None, package_analyzer_version: str = None, package_analyzer_name: str = None, debug: bool = None, package_analyzer_error: bool = None, duration: int = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single package-analyzer (package analysis) run.

debug
duration
package_analysis_datetime
package_analysis_document_id
package_analyzer_error
package_analyzer_name
package_analyzer_version
class thoth.storages.graph.models.PackageExtractRun(uid: int = None, *, analysis_document_id: str = None, analysis_datetime: datetime.datetime = None, package_extract_version: str = None, package_extract_name: str = None, environment_type: str = None, origin: str = None, debug: bool = None, package_extract_error: bool = None, image_tag: str = None, duration: int = None, os_name: str = None, os_id: str = None, os_version_id: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single package-extract (image analysis) run.

analysis_datetime
analysis_document_id
debug
duration
environment_type
image_tag
origin
os_id
os_name
os_version_id
package_extract_error
package_extract_name
package_extract_version
class thoth.storages.graph.models.ProvenanceCheckerRun(uid: int = None, *, provenance_checker_document_id: str = None, provenance_checker_datetime: datetime.datetime = None, provenance_checker_version: str = None, provenance_checker_name: str = None, origin: str = None, debug: bool = None, provenance_checker_error: bool = None, duration: int = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single provenance-checker run.

debug
duration
origin
provenance_checker_datetime
provenance_checker_document_id
provenance_checker_error
provenance_checker_name
provenance_checker_version
class thoth.storages.graph.models.ProvenanceCheckerStackInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

A stack input to a provenance checker run.

class thoth.storages.graph.models.ProvidedBy(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given PythonPackageVersion is provided by a Python index.

class thoth.storages.graph.models.PythonArtifact(uid: int = None, *, artifact_hash_sha256: str = None, artifact_name: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

An artifact for a python package in a specific version.

artifact_hash_sha256
artifact_name
class thoth.storages.graph.models.PythonFileDigest(uid: int = None, *, sha256: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A class representing a single file digests.

sha256
class thoth.storages.graph.models.PythonPackageIndex(uid: int = None, *, url: str = None, warehouse_api_url: str = None, verify_ssl: bool = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

Representation of a Python package Index.

url
verify_ssl
warehouse_api_url
class thoth.storages.graph.models.PythonPackageRequirement(uid: int = None, *, ecosystem: str = None, package_name: str = None, version_range: str = None, develop: bool = None, index_url: str = None, extras: str = None, markers: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

Python package requirement vertex.

develop
ecosystem
extras
index_url
markers
package_name
version_range
class thoth.storages.graph.models.PythonPackageVersion(uid: int = None, *, ecosystem: str = None, package_name: str = None, package_version: str = None, index_url: str = None, extras: str = None, os_name: str = None, os_version: str = None, python_version: str = None, solver_error: bool = None, solver_error_unparseable: bool = None, solver_error_unsolvable: bool = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

Python package version vertex.

ecosystem
extras
index_url
os_name
os_version
package_name
package_version
python_version
solver_error
solver_error_unparseable
solver_error_unsolvable
class thoth.storages.graph.models.PythonPackageVersionEntity(uid: int = None, *, ecosystem: str = None, package_name: str = None, package_version: str = None, index_url: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A representative for a Python package version vertex without being installed.

ecosystem
index_url
package_name
package_version
class thoth.storages.graph.models.RPMPackageVersion(uid: int = None, *, ecosystem: str = None, package_name: str = None, package_version: str = None, release: str = None, epoch: str = None, arch: str = None, src: bool = None, package_identifier: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

RPM-specific package version.

ELEMENT_NAME = 'rpm_package_version'
arch
ecosystem
epoch
package_identifier
package_name
package_version
release
src
class thoth.storages.graph.models.RPMRequirement(uid: int = None, *, rpm_requirement_name: str = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

Requirement of an RPM as stated in a spec file.

ELEMENT_NAME = 'rpm_requirement'
rpm_requirement_name
class thoth.storages.graph.models.RequirementsInput(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Requirement input - Python package requirements.

class thoth.storages.graph.models.Requires(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Requirement edge of an RPM package.

class thoth.storages.graph.models.Resolved(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given dependency monkey run resolved an inspection software stack which was sent to inspection run.

class thoth.storages.graph.models.RunSoftwareEnvironment(uid: int = None, *, environment_name: str = None, python_version: str = None, image_name: str = None, image_sha: str = None, os_name: str = None, os_version: str = None, cuda_version: str = None)[source]

Bases: thoth.storages.graph.models.EnvironmentBase

Software Environment for run - container image which consists of various packages used in a deployment.

cuda_version
class thoth.storages.graph.models.SoftwareStackBase(uid: int = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A software stack crated by packages in specific versions.

This is just a base class for creating software stack instances inside graph database. See specific software stack types for specific software stacks we are interested in.

class thoth.storages.graph.models.Solved(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

The given ecosystem solver solved a stack.

class thoth.storages.graph.models.UsedIn(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Information about hardware used when running/checking a stack.

class thoth.storages.graph.models.UsedInBuild(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Information about hardware used when building a stack on Amun.

class thoth.storages.graph.models.UsedInJob(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

Information about hardware used when running a stack on Amun.

class thoth.storages.graph.models.UserHardwareInformation(uid: int = None, *, cpu_vendor: str = None, cpu_model: int = None, cpu_cores: int = None, cpu_model_name: str = None, cpu_family: int = None, cpu_physical_cpus: int = None, gpu_model_name: str = None, gpu_vendor: str = None, gpu_cores: int = None, gpu_memory_size: float = None, ram_size: float = None)[source]

Bases: thoth.storages.graph.models.HardwareInformation

Hardware specification and properties as used by a user (input for the recommendation engine).

class thoth.storages.graph.models.UserRunSoftwareEnvironment(uid: int = None, *, environment_name: str = None, python_version: str = None, image_name: str = None, image_sha: str = None, os_name: str = None, os_version: str = None, cuda_version: str = None)[source]

Bases: thoth.storages.graph.models.RunSoftwareEnvironment

Software Environment for run - container image which consists of various packages used in a deployment.

As used by a user (input for the recommendation engine).

class thoth.storages.graph.models.UserSoftwareStack(uid: int = None, *, document_id: str = None)[source]

Bases: thoth.storages.graph.models.SoftwareStackBase

A software stack as used by a user (input for the recommendation engine).

document_id

thoth.storages.graph.models_base module

Base operations on top of Dgraph’s models.

class thoth.storages.graph.models_base.EdgeBase(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.Element

A base class for implementing edges.

classmethod from_properties(*, source, target, **properties)[source]

Instantiate a vertex from properties.

get_or_create(client: pydgraph.client.DgraphClient) → bool[source]

Get or create the given edge.

source
target
class thoth.storages.graph.models_base.Element(uid: int = None)[source]

Bases: object

An abstract class with common methods for vertex and edge.

ELEMENT_NAME = None
static compute_label_hash(data) → str[source]

Get hash of this element.

delete(client: pydgraph.client.DgraphClient) → None[source]

Delete the given element from graph database.

classmethod get_label() → str[source]

Retrieve label of the given vertex or edge.

classmethod get_name() → str[source]

Get name of edge.

get_or_create(client: pydgraph.client.DgraphClient) → bool[source]

Get or create the given entity.

classmethod get_properties() → dict[source]

Get all properties with their corresponding type.

classmethod modify_all(client: pydgraph.client.DgraphClient, *, properties: dict, raise_on_not_found: bool = False, **keys) → List[thoth.storages.graph.models_base.Element][source]

Modify all properties of models stored inside graph database matching the given keys.

classmethod modify_one(client: pydgraph.client.DgraphClient, *, properties: dict, raise_on_not_found: bool = False, **keys) → thoth.storages.graph.models_base.Element[source]

Modify all properties of models stored inside graph database matching the given keys.

classmethod query_all(client: pydgraph.client.DgraphClient, *, raise_on_not_found: bool = False, **keys) → List[thoth.storages.graph.models_base.Element][source]

Query graph database for entity with the given set of parameters.

classmethod query_one(client: pydgraph.client.DgraphClient, *, raise_on_not_found: bool = False, **keys) → thoth.storages.graph.models_base.Element[source]

Query graph database for entity with the given set of parameters for an entity.

to_dict(*, without_uid: bool = False) → dict[source]

Covert an edge or vertex representation to a dict.

uid
class thoth.storages.graph.models_base.ReverseEdgeBase(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None)[source]

Bases: thoth.storages.graph.models_base.EdgeBase

An edge which stores also reverse direction.

class thoth.storages.graph.models_base.VertexBase(uid: int = None)[source]

Bases: thoth.storages.graph.models_base.Element

A base class for implementing vertexes.

classmethod from_properties(**properties)[source]

Instantiate a vertex from properties.

get_or_create(client: pydgraph.client.DgraphClient) → bool[source]

Get or create the given vertex.

This is implementation of the upsert procedure.

thoth.storages.graph.models_base.enable_vertex_cache(func: Callable)[source]

Enable vertex caching.

thoth.storages.graph.models_base.model_property(type, default=None, index=None, reverse=None, count=None)[source]

Define a property with its attributes.

Make sure metadata get stored for later automatic schema generation.

thoth.storages.graph.performance module

Performance indicators (models) used in Thoth with their schemas.

class thoth.storages.graph.performance.ObservedPerformance(uid: int = None, *, source: thoth.storages.graph.models_base.VertexBase = None, target: thoth.storages.graph.models_base.VertexBase = None, performance_indicator_index: int = None)[source]

Bases: thoth.storages.graph.models_base.ReverseEdgeBase

A class for representing connection to performance indicators.

performance_indicator_index
class thoth.storages.graph.performance.PerformanceIndicatorBase(uid: int = None, *, framework: str = None, origin: str = None, version: str = None, overall_score: float = None, exit_code: int = None, ru_utime: float = None, ru_stime: float = None, ru_maxrss: int = None, ru_ixrss: int = None, ru_idrss: int = None, ru_isrss: int = None, ru_minflt: int = None, ru_majflt: int = None, ru_nswap: int = None, ru_inblock: int = None, ru_oublock: int = None, ru_msgsnd: int = None, ru_msgrcv: int = None, ru_nsignals: int = None, ru_nvcsw: int = None, ru_nivcsw: int = None)[source]

Bases: thoth.storages.graph.models_base.VertexBase

A base class for implementing performance indicators.

SCHEMA_PARAMETERS = None
SCHEMA_RESULT = None
classmethod create_from_report(inspection_document: dict) → thoth.storages.graph.performance.PerformanceIndicatorBase[source]

Create performance indicator record together with related observed performance edge based on inspection.

exit_code
framework
classmethod from_report(inspection_document: dict, partial_model: type) → thoth.storages.graph.performance.PerformanceIndicatorBase[source]

Create model from the inspection report respecting parameters and result reported by the indicator.

origin
overall_score
ru_idrss
ru_inblock
ru_isrss
ru_ixrss
ru_majflt
ru_maxrss
ru_minflt
ru_msgrcv
ru_msgsnd
ru_nivcsw
ru_nsignals
ru_nswap
ru_nvcsw
ru_oublock
ru_stime
ru_utime
version
class thoth.storages.graph.performance.PiConv1D(uid: int = None, *, framework: str = None, origin: str = None, version: str = None, overall_score: float = None, exit_code: int = None, ru_utime: float = None, ru_stime: float = None, ru_maxrss: int = None, ru_ixrss: int = None, ru_idrss: int = None, ru_isrss: int = None, ru_minflt: int = None, ru_majflt: int = None, ru_nswap: int = None, ru_inblock: int = None, ru_oublock: int = None, ru_msgsnd: int = None, ru_msgrcv: int = None, ru_nsignals: int = None, ru_nvcsw: int = None, ru_nivcsw: int = None, device: str = None, dtype: str = None, reps: int = None, data_format: str = None, batch: int = None, input_width: int = None, input_channels: int = None, filter_width: int = None, output_channels: int = None, strides: int = None, padding: str = None, elapsed: float = None, rate: float = None)[source]

Bases: thoth.storages.graph.performance.PerformanceIndicatorBase

A class for representing a conv1D micro-performance test.

SCHEMA_PARAMETERS = <Schema({'dtype': <class 'str'>, 'reps': <class 'int'>, 'device': <class 'str'>, 'data_format': <class 'str'>, 'batch': <class 'int'>, 'input_width': <class 'int'>, 'input_channels': <class 'int'>, 'filter_width': <class 'int'>, 'output_channels': <class 'int'>, 'strides': <class 'int'>, 'padding': <class 'str'>}, extra=PREVENT_EXTRA, required=False) object>
SCHEMA_RESULT = <Schema({'elapsed': <class 'float'>, 'rate': <class 'float'>}, extra=PREVENT_EXTRA, required=False) object>
batch
data_format
device
dtype
elapsed
filter_width
input_channels
input_width
output_channels
padding
rate
reps
strides
class thoth.storages.graph.performance.PiConv2D(uid: int = None, *, framework: str = None, origin: str = None, version: str = None, overall_score: float = None, exit_code: int = None, ru_utime: float = None, ru_stime: float = None, ru_maxrss: int = None, ru_ixrss: int = None, ru_idrss: int = None, ru_isrss: int = None, ru_minflt: int = None, ru_majflt: int = None, ru_nswap: int = None, ru_inblock: int = None, ru_oublock: int = None, ru_msgsnd: int = None, ru_msgrcv: int = None, ru_nsignals: int = None, ru_nvcsw: int = None, ru_nivcsw: int = None, device: str = None, dtype: str = None, reps: int = None, data_format: str = None, batch: int = None, input_height: int = None, input_width: int = None, input_channels: int = None, filter_height: int = None, filter_width: int = None, output_channels: int = None, strides: int = None, padding: str = None, elapsed: float = None, rate: float = None)[source]

Bases: thoth.storages.graph.performance.PerformanceIndicatorBase

A class for representing a conv2D micro-performance test.

SCHEMA_PARAMETERS = <Schema({'dtype': <class 'str'>, 'reps': <class 'int'>, 'device': <class 'str'>, 'data_format': <class 'str'>, 'batch': <class 'int'>, 'input_height': <class 'int'>, 'input_width': <class 'int'>, 'input_channels': <class 'int'>, 'filter_height': <class 'int'>, 'filter_width': <class 'int'>, 'output_channels': <class 'int'>, 'strides': <class 'int'>, 'padding': <class 'str'>}, extra=PREVENT_EXTRA, required=False) object>
SCHEMA_RESULT = <Schema({'elapsed': <class 'float'>, 'rate': <class 'float'>}, extra=PREVENT_EXTRA, required=False) object>
batch
data_format
device
dtype
elapsed
filter_height
filter_width
input_channels
input_height
input_width
output_channels
padding
rate
reps
strides
class thoth.storages.graph.performance.PiMatmul(uid: int = None, *, framework: str = None, origin: str = None, version: str = None, overall_score: float = None, exit_code: int = None, ru_utime: float = None, ru_stime: float = None, ru_maxrss: int = None, ru_ixrss: int = None, ru_idrss: int = None, ru_isrss: int = None, ru_minflt: int = None, ru_majflt: int = None, ru_nswap: int = None, ru_inblock: int = None, ru_oublock: int = None, ru_msgsnd: int = None, ru_msgrcv: int = None, ru_nsignals: int = None, ru_nvcsw: int = None, ru_nivcsw: int = None, device: str = None, matrix_size: int = None, dtype: str = None, reps: int = None, elapsed: float = None, rate: float = None)[source]

Bases: thoth.storages.graph.performance.PerformanceIndicatorBase

A class for representing a matrix multiplication micro-performance test.

SCHEMA_PARAMETERS = <Schema({'matrix_size': <class 'int'>, 'dtype': <class 'str'>, 'reps': <class 'int'>, 'device': <class 'str'>}, extra=PREVENT_EXTRA, required=False) object>
SCHEMA_RESULT = <Schema({'elapsed': <class 'float'>, 'rate': <class 'float'>}, extra=PREVENT_EXTRA, required=False) object>
device
dtype
elapsed
matrix_size
rate
reps

Module contents

A graph database adapter for communicating with dgraph via gRPC.