thoth.adviser package¶
Subpackages¶
- thoth.adviser.boots package
- Submodules
- thoth.adviser.boots.fully_specified_environment module
- thoth.adviser.boots.pipfile_hash module
- thoth.adviser.boots.platform module
- thoth.adviser.boots.python_version module
- thoth.adviser.boots.rhel_version module
- thoth.adviser.boots.solved_software_environment module
- thoth.adviser.boots.ubi module
- Module contents
- thoth.adviser.predictors package
- Submodules
- thoth.adviser.predictors.annealing module
- thoth.adviser.predictors.hill_climbing module
- thoth.adviser.predictors.latest module
- thoth.adviser.predictors.mcts module
- thoth.adviser.predictors.package_combinations module
- thoth.adviser.predictors.random_walk module
- thoth.adviser.predictors.sampling module
- thoth.adviser.predictors.td module
- Module contents
- thoth.adviser.pseudonyms package
- thoth.adviser.sieves package
- Subpackages
- Submodules
- thoth.adviser.sieves.abi_compat module
- thoth.adviser.sieves.filter_index module
- thoth.adviser.sieves.index_enabled module
- thoth.adviser.sieves.locked module
- thoth.adviser.sieves.prereleases module
- thoth.adviser.sieves.solved module
- thoth.adviser.sieves.version_constraint module
- Module contents
- thoth.adviser.steps package
- Subpackages
- thoth.adviser.steps.tensorflow package
- Submodules
- thoth.adviser.steps.tensorflow.tf_113_numpy module
- thoth.adviser.steps.tensorflow.tf_114_gast module
- thoth.adviser.steps.tensorflow.tf_21_h5py module
- thoth.adviser.steps.tensorflow.tf_21_urllib3 module
- thoth.adviser.steps.tensorflow.tf_22_numpy module
- thoth.adviser.steps.tensorflow.tf_22_prob module
- thoth.adviser.steps.tensorflow.tf_avx2 module
- thoth.adviser.steps.tensorflow.tf_rm_scipy module
- Module contents
- thoth.adviser.steps.tensorflow package
- Submodules
- thoth.adviser.steps.aicoe module
- thoth.adviser.steps.cve module
- thoth.adviser.steps.dropout module
- thoth.adviser.steps.security_indicators module
- Module contents
- Subpackages
- thoth.adviser.strides package
- thoth.adviser.wraps package
- Subpackages
- thoth.adviser.wraps.python package
- thoth.adviser.wraps.pytorch package
- thoth.adviser.wraps.tensorflow package
- Submodules
- thoth.adviser.wraps.tensorflow.tf_23_accuracy module
- thoth.adviser.wraps.tensorflow.tf_23_dict_bug module
- thoth.adviser.wraps.tensorflow.tf_38518 module
- thoth.adviser.wraps.tensorflow.tf_42475 module
- thoth.adviser.wraps.tensorflow.tf_intel module
- thoth.adviser.wraps.tensorflow.tf_mkl_threads module
- Module contents
- Submodules
- thoth.adviser.wraps.no_onservation module
- Module contents
- Subpackages
Submodules¶
thoth.adviser.beam module¶
Implementation of Beam for beam searching part of adviser.
-
class
thoth.adviser.beam.
Beam
(width: Optional[int] = None, *, keep_history=None, beam_history: List[Tuple[int, Optional[float]]] = NOTHING)[source]¶ Bases:
object
Beam implementation.
The implementation of beam respecting beam width: https://en.wikipedia.org/wiki/Beam_search
Beam is internally implemented on top of a min-heap queue to optimize inserts and respect beam width in O(log(N)).
The most frequent operation performed on the beam is not always the same - it depends on the beam width and number of states generated. If number of states generated is more than the width of the beam, its reasonable to optimize insertions into the beam with checks on beam width.
If number of states generated is smaller than the beam width, the beam could be optimized for removal of states.
As the first scenario is seen in a real-world deployment, the beam uses min-heapq to keep addition to the beam with beam_width checks in O(log(N)) and removals of the states in O(log(N)). To satisfy removals in O(log(N)), the beam maintains a dictionary mapping a state to its index in the beam.
-
add_state
(state: thoth.adviser.state.State) → None[source]¶ Add state to the internal state listing (do it in O(log(N)) time.
-
get
(idx: int) → thoth.adviser.state.State[source]¶ Get i-th element from the beam (constant time), keep it in the beam.
This method is suitable for random state indexing in the beam (like in case of adaptive simulated annealing). The actual index is not into a sorted array and has no special meaning assigned - beam under the hood uses min-heapq (as of now), but the index used is not guaranteed to point to a heap-like data structure.
-
get_last
() → Optional[thoth.adviser.state.State][source]¶ Get state that was added in the previous resolution round.
-
iter_states
() → List[thoth.adviser.state.State][source]¶ Iterate over states, do not respect their score in order of iteration.
-
iter_states_sorted
(reverse: bool = True) → Generator[thoth.adviser.state.State, None, None][source]¶ Iterate over sorted states.
-
keep_history
¶
-
new_iteration
() → None[source]¶ Called once a new iteration is done in resolver.
Used to keep track of beam history and to keep track of states added.
-
plot
() → matplotlib.figure.Figure[source]¶ Plot temperature history of adaptive simulated annealing.
-
pop
(idx: Optional[int] = None) → thoth.adviser.state.State[source]¶ Pop i-th element from the beam and remove it from the beam (this is actually toppop).
If index is not provided, pop the largest item kept in the beam.
-
property
size
¶ Get the current size of beam.
-
width
¶
-
thoth.adviser.boot module¶
A base class for implementing boot units.
-
class
thoth.adviser.boot.
Boot
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Boot base class implementation.
thoth.adviser.cli module¶
Thoth-adviser CLI.
thoth.adviser.context module¶
Pipeline context carried during annealing.
-
class
thoth.adviser.context.
Context
(*, project: thoth.python.project.Project, graph: thoth.storages.graph.postgres.GraphDatabase, library_usage: Optional[Dict[str, Any]], limit: int, count: int, beam: thoth.adviser.beam.Beam, recommendation_type: Optional[thoth.adviser.enums.RecommendationType] = None, decision_type: Optional[thoth.adviser.enums.DecisionType] = None, package_versions: Dict[Tuple[str, str, str], thoth.python.package_version.PackageVersion] = NOTHING, dependencies: Dict[str, Dict[Tuple[str, str, str], Set[Tuple[str, str, str]]]] = NOTHING, dependents: Dict[str, Dict[Tuple[str, str, str], Set[Tuple[Tuple[str, str, str], Optional[str], Optional[str], Optional[str]]]]] = NOTHING, sources: Dict[str, thoth.python.source.Source] = NOTHING, iteration: int = 0, cli_parameters: Dict[str, Any] = NOTHING, stack_info: List[Dict[str, Any]] = NOTHING, accepted_final_states_count: int = 0, discarded_final_states_count: int = 0, accepted_states: List[Tuple[Tuple[float, int], thoth.adviser.state.State]] = NOTHING, accepted_states_counter: int = 0)[source]¶ Bases:
object
Context carried during adviser’s pipeline run.
It’s suitable to cache entries such as PackageVersion to optimize memory usage and optimize overhead needed - for example for parsing version strings (this is lazily pre-cached in PackageVersion).
-
accepted_final_states_count
¶
-
beam
¶
-
cli_parameters
¶
-
count
¶
-
decision_type
¶
-
dependencies
¶
-
dependents
¶
-
discarded_final_states_count
¶
-
get_package_version
(package_tuple: Tuple[str, str, str], *, graceful: bool = False) → Optional[thoth.python.package_version.PackageVersion][source]¶ Get the given package version registered to the context.
-
get_top_accepted_final_state
() → Optional[thoth.adviser.state.State][source]¶ Get the best accepted final state so far computed by the resolution pipeline.
-
graph
¶
-
is_dependency_monkey
() → bool[source]¶ Check if the current context refers to a dependency monkey run.
-
iter_accepted_final_states
() → Generator[thoth.adviser.state.State, None, None][source]¶ Get accepted final states by resolution pipeline, states are not sorted.
-
iter_accepted_final_states_sorted
(reverse: bool = True) → Generator[thoth.adviser.state.State, None, None][source]¶ Get accepted final states by resolution pipeline sorted by score and their precedence.
-
iteration
¶
-
library_usage
¶
-
limit
¶
-
package_versions
¶
-
project
¶
-
recommendation_type
¶
-
register_accepted_final_state
(state: thoth.adviser.state.State) → None[source]¶ Register an accepted state by the resolution pipeline.
-
register_package_tuple
(package_tuple: Tuple[str, str, str], *, develop: bool, dependent_tuple: Optional[Tuple[str, str, str]] = None, extras: Optional[List[str]] = None, os_name: Optional[str], os_version: Optional[str], python_version: Optional[str]) → thoth.python.package_version.PackageVersion[source]¶ Register the given package tuple to pipeline context and return its package version representative.
-
register_package_version
(package_version: thoth.python.package_version.PackageVersion) → bool[source]¶ Register the given package version to the context.
-
sources
¶
-
stack_info
¶
-
thoth.adviser.dependency_monkey module¶
Routines for dependency monkey and its output handling.
-
class
thoth.adviser.dependency_monkey.
DependencyMonkey
(*, resolver: thoth.adviser.resolver.Resolver, stack_output: str = '-', context: Optional[Dict[Any, Any]] = NOTHING, dry_run: bool = False, decision_type: thoth.adviser.enums.DecisionType = <DecisionType.ALL: 1>)[source]¶ Bases:
object
Dependency monkey creates stacks based on the configuration using ASA.
-
property
beam
¶ Get beam instance used in the resolver.
-
context
¶
-
decision_type
¶
-
dry_run
¶
-
property
predictor
¶ Get predictor for the current dependency monkey configuration.
-
resolve
(*, with_devel: bool = True, user_stack_scoring: bool = False) → thoth.adviser.dm_report.DependencyMonkeyReport[source]¶ Perform simulated annealing and run dependency monkey on products.
-
resolver
¶
-
stack_output
¶
-
property
thoth.adviser.digests_fetcher module¶
Fetcher for fetching digests from the graph database.
-
class
thoth.adviser.digests_fetcher.
GraphDigestsFetcher
(graph: thoth.storages.graph.postgres.GraphDatabase = NOTHING)[source]¶ Bases:
thoth.python.digests_fetcher.DigestsFetcherBase
Fetch digests from the graph database.
-
fetch_digests
(package_name: str, package_version: str) → Dict[str, List[Dict[str, str]]][source]¶ Fetch digests for the given package in specified version, consider only enabled indexes.
-
graph
¶
-
thoth.adviser.dm_report module¶
Routines for dependency monkey and its output handling.
-
class
thoth.adviser.dm_report.
DependencyMonkeyReport
(skipped: int = 0, responses: List[Dict[str, Union[Dict[str, Any], str]]] = NOTHING)[source]¶ Bases:
object
Report produced by a Dependency Monkey run.
-
add_response
(response: str, product: thoth.adviser.product.Product) → None[source]¶ Add a new response to response listing.
-
skipped
¶
-
thoth.adviser.enums module¶
Enum types used in thoth-adviser code.
-
class
thoth.adviser.enums.
DecisionType
[source]¶ Bases:
thoth.adviser.enums._ExtendedEnum
Type of decision used in Dependency Monkey for generating stacks.
-
ALL
= 1¶
-
RANDOM
= 2¶
-
-
class
thoth.adviser.enums.
Ecosystem
[source]¶ Bases:
thoth.adviser.enums._ExtendedEnum
Ecosystem known to recommendation engine.
-
PYTHON
= 1¶
-
thoth.adviser.exceptions module¶
Exception hierarchy used in the whole adviser implementation.
-
exception
thoth.adviser.exceptions.
AdviserException
[source]¶ Bases:
Exception
A base adviser exception in adviser’s exception hierarchy.
-
exception
thoth.adviser.exceptions.
AdviserPipelineException
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
A base class for implementing pipeline specific exceptions.
-
exception
thoth.adviser.exceptions.
AdviserRunException
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
A base class for implementing exceptions occurred during an andviser run.
-
exception
thoth.adviser.exceptions.
BootError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when pipeline boot unit fails unexpectedly.
-
exception
thoth.adviser.exceptions.
CannotProduceStack
(*args: Any, stack_info: List[Dict[str, Any]])[source]¶ Bases:
thoth.adviser.exceptions.AdviserRunException
Raised if there was not produced any result.
-
stack_info
¶
-
-
exception
thoth.adviser.exceptions.
EagerStopPipeline
[source]¶ Bases:
thoth.adviser.exceptions.AdviserPipelineException
Raised to signalize premature end of pipeline.
-
exception
thoth.adviser.exceptions.
InternalError
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
An exception raised on bugs in the code base.
-
exception
thoth.adviser.exceptions.
NoHistoryKept
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
Raised if a user asks for history, but history was not kept (e.g. temperature function history in annealing).
-
exception
thoth.adviser.exceptions.
NotAcceptable
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
An exception raised during stack generation when the given action would produce un-acceptable solution.
-
exception
thoth.adviser.exceptions.
NotFound
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
An exception raised if the given resource was not found.
-
exception
thoth.adviser.exceptions.
PipelineConfigurationError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when a wrong pipeline unit configuration supplied.
Or any error during configuration initialization.
-
exception
thoth.adviser.exceptions.
PipelineUnitConfigurationSchemaError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when pipeline unit configuration does not match schema declared.
-
exception
thoth.adviser.exceptions.
PipelineUnitError
[source]¶ Bases:
thoth.adviser.exceptions.AdviserPipelineException
An exception raised when there is an error during pipeline run, unexpectedly.
-
exception
thoth.adviser.exceptions.
PipfileParseError
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
An exception raised on invalid Pipfile or Pipfile.lock.
-
exception
thoth.adviser.exceptions.
SieveError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when pipeline sieve unit fails unexpectedly.
-
exception
thoth.adviser.exceptions.
SkipPackage
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
An exception raised during stack generation when the given package should be skipped in the stack.
-
exception
thoth.adviser.exceptions.
StepError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when pipeline step unit fails unexpectedly.
-
exception
thoth.adviser.exceptions.
StrideError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when pipeline stride unit fails unexpectedly.
-
exception
thoth.adviser.exceptions.
UnableLock
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
Raised if it is unable to lock dependencies given the set of constraints.
-
exception
thoth.adviser.exceptions.
UnknownPipelineUnitError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when an unknown pipeline unit is requested.
-
exception
thoth.adviser.exceptions.
UnresolvedDependencies
(*args: Any, unresolved: List[str], stack_info: List[Dict[str, Any]])[source]¶ Bases:
thoth.adviser.exceptions.AdviserRunException
An exception raised if dependencies were not resolved and cannot produce stack.
-
stack_info
¶
-
to_dict
() → Optional[Dict[str, Any]][source]¶ Convert unresolved dependencies exception to the user.
-
unresolved
¶
-
-
exception
thoth.adviser.exceptions.
UnsupportedConfiguration
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
An exception raised on unsupported configuration by recommendation engine.
-
exception
thoth.adviser.exceptions.
UserLockFileError
[source]¶ Bases:
thoth.adviser.exceptions.AdviserRunException
An exception raised when the supplied user stack has issues.
-
exception
thoth.adviser.exceptions.
VersionIdentifierError
[source]¶ Bases:
thoth.adviser.exceptions.AdviserException
An exception raised if the given version identifier is not a semver identifier.
-
exception
thoth.adviser.exceptions.
WrapError
[source]¶ Bases:
thoth.adviser.exceptions.PipelineUnitError
An exception raised when pipeline stride unit fails unexpectedly.
thoth.adviser.pipeline_builder module¶
Implementation of pipeline builder - create pipeline configuration based on the project and its library usage.
-
class
thoth.adviser.pipeline_builder.
PipelineBuilder
[source]¶ Bases:
object
Builder responsible for creating pipeline configuration from the project and its library usage.
-
classmethod
from_dict
(dict_: Dict[str, Any]) → thoth.adviser.pipeline_config.PipelineConfig[source]¶ Instantiate pipeline configuration based on dictionary supplied.
-
classmethod
get_adviser_pipeline_config
(*, recommendation_type: thoth.adviser.enums.RecommendationType, graph: thoth.storages.graph.postgres.GraphDatabase, project: thoth.python.project.Project, library_usage: Optional[Dict[str, Any]]) → thoth.adviser.pipeline_config.PipelineConfig[source]¶ Get adviser’s pipeline configuration.
-
classmethod
get_dependency_monkey_pipeline_config
(*, decision_type: thoth.adviser.enums.DecisionType, graph: thoth.storages.graph.postgres.GraphDatabase, project: thoth.python.project.Project, library_usage: Optional[Dict[str, Any]]) → thoth.adviser.pipeline_config.PipelineConfig[source]¶ Get dependency-monkey’s pipeline configuration.
-
classmethod
-
class
thoth.adviser.pipeline_builder.
PipelineBuilderContext
(*, graph: thoth.storages.graph.postgres.GraphDatabase = None, project: thoth.python.project.Project = None, library_usage: Optional[Dict[str, Any]] = None, decision_type: Optional[thoth.adviser.enums.DecisionType] = None, recommendation_type: Optional[thoth.adviser.enums.RecommendationType] = None, boots: Dict[Optional[str], List[thoth.adviser.boot.Boot]] = NOTHING, pseudonyms: Dict[str, List[thoth.adviser.pseudonym.Pseudonym]] = NOTHING, sieves: Dict[Optional[str], List[thoth.adviser.sieve.Sieve]] = NOTHING, steps: Dict[Optional[str], List[thoth.adviser.step.Step]] = NOTHING, strides: Dict[Optional[str], List[thoth.adviser.stride.Stride]] = NOTHING, wraps: Dict[Optional[str], List[thoth.adviser.wrap.Wrap]] = NOTHING, boots_included: Dict[str, List[thoth.adviser.boot.Boot]] = NOTHING, pseudonyms_included: Dict[str, List[thoth.adviser.pseudonym.Pseudonym]] = NOTHING, sieves_included: Dict[str, List[thoth.adviser.sieve.Sieve]] = NOTHING, steps_included: Dict[str, List[thoth.adviser.step.Step]] = NOTHING, strides_included: Dict[str, List[thoth.adviser.stride.Stride]] = NOTHING, wraps_included: Dict[str, List[thoth.adviser.wrap.Wrap]] = NOTHING)[source]¶ Bases:
object
A context passed to units to determine if they want to participate in a pipeline.
-
add_unit
(unit: thoth.adviser.unit.Unit) → None[source]¶ Add the given unit to pipeline configuration.
-
property
boots
¶ Get all boots registered to this pipeline builder context.
-
property
boots_dict
¶ Get boots as a dictionary mapping.
-
decision_type
¶
-
get_included_boots
(boot_class: type) → List[thoth.adviser.boot.Boot][source]¶ Get included boots of the provided boot class.
-
get_included_pseudonyms
(pseudonym_class: type) → List[thoth.adviser.pseudonym.Pseudonym][source]¶ Get included sieves of the provided sieve class.
-
get_included_sieves
(sieve_class: type) → List[thoth.adviser.sieve.Sieve][source]¶ Get included sieves of the provided sieve class.
-
get_included_steps
(step_class: type) → List[thoth.adviser.step.Step][source]¶ Get included steps of the provided step class.
-
get_included_strides
(stride_class: type) → List[thoth.adviser.stride.Stride][source]¶ Get included strides of the provided stride class.
-
get_included_wraps
(wrap_class: type) → List[thoth.adviser.wrap.Wrap][source]¶ Get included wraps of the provided wrap class.
-
graph
¶
-
is_dependency_monkey_pipeline
() → bool[source]¶ Check if the pipeline built is meant for Dependency Monkey.
-
is_included
(unit_class: type) → bool[source]¶ Check if the given pipeline unit is already included in the pipeline configuration.
-
library_usage
¶
-
project
¶
-
property
pseudonyms
¶ Get all pseudonyms registered to this pipeline builder context.
-
property
pseudonyms_dict
¶ Get pseudonyms as a dictionary mapping.
-
recommendation_type
¶
-
property
sieves
¶ Get all sieves registered to this pipeline builder context.
-
property
sieves_dict
¶ Get sieves as a dictionary mapping.
-
property
steps
¶ Get all steps registered to this pipeline builder context.
-
property
steps_dict
¶ Get steps as a dictionary mapping.
-
property
strides
¶ Get all strides registered to this pipeline builder context.
-
property
strides_dict
¶ Get strides as a dictionary mapping.
-
property
wraps
¶ Get all wraps registered to this pipeline builder context.
-
property
wraps_dict
¶ Get wraps as a dictionary mapping.
-
thoth.adviser.pipeline_config module¶
A base class for implementing sieves.
-
class
thoth.adviser.pipeline_config.
PipelineConfig
(boots: Dict[Optional[str], List[thoth.adviser.boot.Boot]] = NOTHING, pseudonyms: Dict[str, List[thoth.adviser.pseudonym.Pseudonym]] = NOTHING, sieves: Dict[Optional[str], List[thoth.adviser.sieve.Sieve]] = NOTHING, steps: Dict[Optional[str], List[thoth.adviser.step.Step]] = NOTHING, strides: Dict[Optional[str], List[thoth.adviser.stride.Stride]] = NOTHING, wraps: Dict[Optional[str], List[thoth.adviser.wrap.Wrap]] = NOTHING)[source]¶ Bases:
object
A configuration of a pipeline for dependency-monkey and for adviser.
-
property
boots
¶ Get all boots.
-
property
boots_dict
¶ Get boots as a dictionary mapping.
-
property
pseudonyms
¶ Get all pseudonyms.
-
property
pseudonyms_dict
¶ Get pseudonyms as a dictionary mapping.
-
property
sieves
¶ Get all sieves.
-
property
sieves_dict
¶ Get sieves as a dictionary mapping.
-
property
steps
¶ Get all steps.
-
property
steps_dict
¶ Get steps as a dictionary mapping.
-
property
strides
¶ Get all strides.
-
property
strides_dict
¶ Get strides as a dictionary mapping.
-
to_dict
() → Dict[str, List[Dict[str, Any]]][source]¶ Return this pipeline configuration in a dict representation.
-
property
wraps
¶ Get all wraps.
-
property
wraps_dict
¶ Get wraps as a dictionary mapping.
-
property
thoth.adviser.predictor module¶
A base class for implementing predictor.
-
class
thoth.adviser.predictor.
Predictor
(*, keep_history=None)[source]¶ Bases:
object
A base class for implementing a predictor for resolver.
-
classmethod
assigned_context
(context: thoth.adviser.context.Context) → Generator[None, None, None][source]¶ Assign context to predictor.
-
property
context
¶ Get context in which the unit runs in.
-
finalize_state
(state_id: int) → None[source]¶ Finalizer called when the given state is about to be destructed by garbage collector.
Method suitable if predictor keeps internal state for states. Note that this method is not called for remaining states if the resolver terminates.
@param state_id: id of state that is about to be finalized
-
keep_history
¶
-
classmethod
obtain_default_configuration
(config_option_name: str) → Any[source]¶ Get the default value assigned to the given configuration option of a predictor instance.
-
post_run_report
(report: thoth.adviser.report.Report) → None[source]¶ Post-run method run after the resolving has finished - this method is called only if resolving with a report.
The default operation is a noop, but a predictor can perform any post-run operations in this method. This method should not raise any exception.
-
pre_run
() → None[source]¶ Pre-initialize the predictor.
This method is called before any resolving with a freshly instantiated context. The default operation is a noop, but predictor can perform any initial setup in this method. This method should not raise any exception.
-
abstract
run
() → Tuple[thoth.adviser.state.State, Tuple[str, str, str]][source]¶ Run the main method used to run the predictor.
-
set_reward_signal
(state: thoth.adviser.state.State, package_tuple: Tuple[str, str, str], reward: float) → None[source]¶ Signalize the reward.
@param state: (child) state for which the reward signal is triggered @param package_tuple: Python package that was added to the state causing the reward @param reward: set to nan if the given state was not accepted a special value
of inf notifies about a new final state
-
classmethod
thoth.adviser.product module¶
Representation of an advised stack.
-
class
thoth.adviser.product.
Product
(*, project: thoth.python.project.Project, score: float, justification: List[Dict[str, str]], advised_runtime_environment: Optional[thoth.common.config.runtime_environment.RuntimeEnvironment] = None, advised_manifest_changes: List[List[Dict[str, Any]]] = NOTHING)[source]¶ Bases:
object
A representation of an advised stack.
-
advised_manifest_changes
¶
-
advised_runtime_environment
¶
-
classmethod
from_final_state
(*, context: thoth.adviser.context.Context, state: thoth.adviser.state.State) → thoth.adviser.product.Product[source]¶ Instantiate advised stack from final state produced by adviser’s pipeline.
-
justification
¶
-
project
¶
-
score
¶
-
thoth.adviser.pseudonym module¶
A base class for implementing package pseudonyms.
-
class
thoth.adviser.pseudonym.
Pseudonym
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Pseudonym base class implementation.
-
CONFIGURATION_SCHEMA
= <Schema({'package_name': <class 'str'>}, extra=PREVENT_EXTRA, required=False) object>¶
-
thoth.adviser.report module¶
A class for adviser’s report - output of an adviser run.
-
class
thoth.adviser.report.
Report
(*, count: int, pipeline: thoth.adviser.pipeline_config.PipelineConfig, resolver_iterations: int = 0, accepted_final_states_count: int = 0, discarded_final_states_count: int = 0, stack_info: Optional[List[Dict[str, Any]]] = None, heapq: List[Tuple[Tuple[float, int], thoth.adviser.product.Product]] = NOTHING, heapq_counter: int = 0)[source]¶ Bases:
object
A report stating output of an adviser run.
-
accepted_final_states_count
¶
-
add_product
(product: thoth.adviser.product.Product) → bool[source]¶ Add adviser pipeline product to report.
-
count
¶
-
discarded_final_states_count
¶
-
iter_products
() → Generator[thoth.adviser.product.Product, None, None][source]¶ Iterate over products stored in this report, respect their scores.
-
iter_products_sorted
(reverse: bool = True) → Generator[thoth.adviser.product.Product, None, None][source]¶ Iterate over products stored in this report, respect their scores.
-
pipeline
¶
-
resolver_iterations
¶
-
property
stack_info
¶ Retrieve stack info of this report.
-
thoth.adviser.resolver module¶
The main resolving algorithm working on top of states.
-
class
thoth.adviser.resolver.
Resolver
(*, pipeline: thoth.adviser.pipeline_config.PipelineConfig, project: thoth.python.project.Project, library_usage, graph: thoth.storages.graph.postgres.GraphDatabase, predictor: thoth.adviser.predictor.Predictor, recommendation_type: Optional[thoth.adviser.enums.RecommendationType] = None, decision_type: Optional[thoth.adviser.enums.DecisionType] = None, limit: int = 10000, count: int = 3, beam_width=-1, limit_latest_versions=-1, cli_parameters: Dict[str, Any] = NOTHING, stop_resolving: bool = False, log_iteration: int = 7500, beam: Optional[thoth.adviser.beam.Beam] = None, solver: Optional[thoth.adviser.solver.PythonPackageGraphSolver] = None, context: Optional[thoth.adviser.context.Context] = None, log_unresolved: Set[Tuple[str, str, str]] = NOTHING, log_unsolved: Set[str] = NOTHING, log_sieved: Set[str] = NOTHING, log_step_skip_package: Set[Tuple[str, str, str]] = NOTHING, log_step_not_acceptable: Set[Tuple[str, str, str]] = NOTHING, log_no_intersected: Set[Tuple[Tuple[str, str, str], str]] = NOTHING)[source]¶ Bases:
object
Resolver for resolving software stacks using pipeline configuration and a predictor.
-
DEFAULT_BEAM_WIDTH
= -1¶
-
DEFAULT_COUNT
= 3¶
-
DEFAULT_LIMIT
= 10000¶
-
DEFAULT_LIMIT_LATEST_VERSIONS
= -1¶
-
property
beam
¶ Get beam for storing states.
-
beam_width
¶
-
cli_parameters
¶
-
property
context
¶ Retrieve context bound to the current resolver.
-
count
¶
-
decision_type
¶
-
classmethod
get_adviser_instance
(*, predictor: thoth.adviser.predictor.Predictor, beam_width: Optional[int] = None, count: int = 3, graph: Optional[thoth.storages.graph.postgres.GraphDatabase] = None, library_usage: Optional[Dict[str, Any]] = None, limit: int = 10000, limit_latest_versions: Optional[int] = -1, project: thoth.python.project.Project, recommendation_type: thoth.adviser.enums.RecommendationType, pipeline_config: Union[thoth.adviser.pipeline_config.PipelineConfig, Dict[str, Any], None] = None, cli_parameters: Optional[Dict[str, Any]] = None) → thoth.adviser.resolver.Resolver[source]¶ Get instance of resolver based on the project given to recommend software stacks.
-
classmethod
get_dependency_monkey_instance
(*, predictor: thoth.adviser.predictor.Predictor, beam_width: Optional[int] = None, count: int = 3, graph: Optional[thoth.storages.graph.postgres.GraphDatabase] = None, library_usage: Optional[Dict[str, Any]] = None, limit_latest_versions: Optional[int] = -1, project: thoth.python.project.Project, decision_type: thoth.adviser.enums.DecisionType, pipeline_config: Union[thoth.adviser.pipeline_config.PipelineConfig, Dict[str, Any], None] = None, cli_parameters: Optional[Dict[str, Any]] = None) → thoth.adviser.resolver.Resolver[source]¶ Get instance of resolver based on the project given to run dependency monkey.
-
graph
¶
-
library_usage
¶
-
limit
¶
-
limit_latest_versions
¶
-
log_iteration
¶
-
pipeline
¶
-
predictor
¶
-
project
¶
-
recommendation_type
¶
-
resolve
(*, with_devel: bool = True, user_stack_scoring: bool = True) → thoth.adviser.report.Report[source]¶ Resolve software stacks and return resolver report.
-
resolve_products
(*, with_devel: bool = True, user_stack_scoring: bool = True) → Generator[thoth.adviser.product.Product, None, None][source]¶ Resolve raw products as produced by this resolver pipeline.
-
property
solver
¶ Get solver instance - solver implemented on top of graph database.
-
stop_resolving
¶
-
thoth.adviser.run module¶
Utilities which wrap adviser’s and dependency monkey’s and run them in a subprocess.
This is useful for a cluster deployment where we restrict resources allocated for an adviser or dependency monkey run. Annealing is run as a subprocess to the main process - if resources are exhausted (CPU time or memory allocated), the subprocess is killed (either by liveness probe or by OOM killer) and the main process can still produce an error message.
-
thoth.adviser.run.
subprocess_run
(resolver: Union[thoth.adviser.resolver.Resolver, thoth.adviser.dependency_monkey.DependencyMonkey], print_func: Callable[[float, Union[Dict[str, Any], List[Any]]], None], result_dict: Dict[str, Any], plot: Optional[str] = None, *, with_devel: bool = True, user_stack_scoring: bool = True) → int[source]¶ Run the given function (partial annealing method) in a subprocess and output the produced report.
thoth.adviser.sieve module¶
A base class for implementing steps.
-
class
thoth.adviser.sieve.
Sieve
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Sieve base class implementation.
thoth.adviser.solver module¶
Definition of package resolution based on precomputed data available in graph.
There are defined primitives required for offline resolution. This off-line resolution is done based on data aggregated in the graph database so thereis no need to perform resolving by actually installing Python packages (as this version resolution is dynamic in case of Python).
-
class
thoth.adviser.solver.
GraphReleasesFetcher
(*, graph: thoth.storages.graph.postgres.GraphDatabase, runtime_environment: thoth.common.config.runtime_environment.RuntimeEnvironment = NOTHING)[source]¶ Bases:
thoth.solver.python.base.ReleasesFetcher
Fetch releases for packages from the graph database.
-
fetch_releases
(package_name: str) → Tuple[str, List[Tuple[str, str]]][source]¶ Fetch releases for the given package name.
-
graph
¶
-
runtime_environment
¶
-
-
class
thoth.adviser.solver.
PackageVersionDependencyParser
[source]¶ Bases:
thoth.solver.python.base.DependencyParser
Parse an instance of PackageVersion to Dependency object needed by solver.
-
class
thoth.adviser.solver.
PythonGraphSolver
(*, dependency_parser: thoth.adviser.solver.PackageVersionDependencyParser, releases_fetcher: thoth.adviser.solver.GraphReleasesFetcher)[source]¶ Bases:
thoth.solver.python.base.Solver
Solve Python dependencies based on data available in the graph database.
-
dependency_parser
¶
-
releases_fetcher
¶
-
-
class
thoth.adviser.solver.
PythonPackageGraphSolver
(*, graph: thoth.storages.graph.postgres.GraphDatabase, runtime_environment: thoth.common.config.runtime_environment.RuntimeEnvironment = NOTHING, package_versions: Dict[Tuple[str, str, str], thoth.python.package_version.PackageVersion] = NOTHING, sources: Dict[str, thoth.python.source.Source] = NOTHING, solver: thoth.adviser.solver.PythonGraphSolver = None)[source]¶ Bases:
object
A wrapper to manipulate with Python packages using pure PackageVersion object interface.
-
graph
¶
-
runtime_environment
¶
-
solve
(dependencies: List[thoth.python.package_version.PackageVersion], graceful: bool = True) → Dict[str, List[thoth.python.package_version.PackageVersion]][source]¶ Solve the given dependencies and return object representation of packages.
-
property
solver
¶ Retrieve solver instance resolving using graph database.
-
thoth.adviser.state module¶
A state of not fully resolved software stack in adviser’s recommendations implementation.
-
class
thoth.adviser.state.
State
(score: float = 0.0, iteration: int = 0, unresolved_dependencies=NOTHING, resolved_dependencies=NOTHING, parent=None, *, advised_runtime_environment: Optional[thoth.common.config.runtime_environment.RuntimeEnvironment] = None, advised_manifest_changes: List[List[Dict[str, Any]]] = NOTHING, justification: List[Dict[str, str]] = NOTHING)[source]¶ Bases:
object
Implementation of an adviser’s state in state space.
-
add_justification
(justification: List[Dict[str, str]]) → None[source]¶ Add new entries to the justification field.
-
add_resolved_dependency
(package_tuple: Tuple[str, str, str]) → None[source]¶ Add a resolved dependency into the state.
-
add_unresolved_dependency
(package_tuple: Tuple[str, str, str]) → None[source]¶ Add unresolved dependency into the state.
-
advised_manifest_changes
¶
-
advised_runtime_environment
¶
-
clone
() → thoth.adviser.state.State[source]¶ Return a swallow copy of this state that can be used as a next state.
-
classmethod
from_direct_dependencies
(direct_dependencies: Dict[str, List[thoth.python.package_version.PackageVersion]]) → thoth.adviser.state.State[source]¶ Create an initial state out of direct dependencies.
-
get_first_unresolved_dependency
(dependency_name: Optional[str] = None) → Tuple[str, str, str][source]¶ Get a very first unresolved dependency tuple.
-
get_random_first_unresolved_dependency
(dependency_name: Optional[str] = None) → Tuple[str, str, str][source]¶ Get a very first unresolved dependency tuple.
-
get_random_unresolved_dependency
(dependency_name: Optional[str] = None, prefer_recent: bool = True) → Tuple[str, str, str][source]¶ Get a very first unresolved dependency tuple.
-
iter_resolved_dependencies
() → Generator[Tuple[str, str, str], None, None][source]¶ Iterate over resolved dependencies.
-
iter_unresolved_dependencies
() → Generator[Tuple[str, str, str], None, None][source]¶ Iterate over unresolved dependencies.
-
iteration
¶
-
justification
¶
-
mark_dependency_resolved
(package_tuple: Tuple[str, str, str]) → None[source]¶ Mark the given dependency as resolved in the current state.
-
property
parent
¶ Retrieve parent to this state.
If None the state is top level state or parent is no longer maintained. Note the return value of None depends on actual gc runs.
-
remove_unresolved_dependency
(package_tuple: Tuple[str, str, str]) → None[source]¶ Remove the given unresolved dependency from state.
-
remove_unresolved_dependency_subtree
(package_name: str) → None[source]¶ Remove the whole dependency sub-tree from the state.
-
resolved_dependencies
¶
-
score
¶
-
set_unresolved_dependencies
(dependencies: Dict[str, List[Tuple[str, str, str]]]) → None[source]¶ Set unresolved dependencies - any unresolved dependencies will be overwritten.
-
unresolved_dependencies
¶
-
thoth.adviser.step module¶
A base class for implementing steps.
-
class
thoth.adviser.step.
Step
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Step base class implementation.
Configuration option mutli_package_resolution states whether a step should be run if package is resolved multiple times for the same stack.
-
CONFIGURATION_DEFAULT
= {'multi_package_resolution': False, 'package_name': None}¶
-
CONFIGURATION_SCHEMA
= <Schema({'package_name': Any(<class 'str'>, None, msg=None), 'multi_package_resolution': <class 'bool'>}, extra=PREVENT_EXTRA, required=False) object>¶
-
SCORE_MAX
= 1.0¶
-
SCORE_MIN
= -1.0¶
-
thoth.adviser.stride module¶
A base class for implementing strides.
-
class
thoth.adviser.stride.
Stride
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Stride base class implementation.
thoth.adviser.unit module¶
A base class for implementing pipeline units - strides and steps.
-
class
thoth.adviser.unit.
Unit
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
object
A base class for implementing pipeline units - strides and steps.
-
CONFIGURATION_DEFAULT
= {'package_name': None}¶
-
CONFIGURATION_SCHEMA
= <Schema({'package_name': Any(<class 'str'>, None, msg=None)}, extra=PREVENT_EXTRA, required=False) object>¶
-
classmethod
assigned_context
(context: thoth.adviser.context.Context) → Generator[None, None, None][source]¶ Assign context to all units.
-
classmethod
compute_expanded_configuration
(configuration_dict: Optional[Dict[str, Any]]) → Dict[str, Any][source]¶ Compute configuration as they would be computed based on unit configuration.
-
property
configuration
¶ Get configuration of instantiated pipeline unit.
-
property
context
¶ Get context in which the unit runs in.
-
classmethod
get_aicoe_configuration
(package_version: thoth.python.package_version.PackageVersion) → Optional[Dict[str, Any]][source]¶ Get AICoE specific configuration encoded in the AICoE index URL.
-
classmethod
is_aicoe_release
(package_version: thoth.python.package_version.PackageVersion) → bool[source]¶ Check if the given package-version is AICoE release.
-
property
name
¶ Get name of this pipeline unit.
-
post_run
() → None[source]¶ Called after the resolution is finished.
This method should not raise any exception.
-
post_run_report
(report)[source]¶ Post-run method run after the resolving has finished - this method is called only if resolving with a report.
This method should not raise any exception.
-
pre_run
() → None[source]¶ Called before running any pipeline unit with context already assigned.
This method should not raise any exception.
-
classmethod
should_include
(builder_context: PipelineBuilderContext) → Optional[Dict[str, Any]][source]¶ Check if the given pipeline unit should be included in the given pipeline configuration.
-
unit_run
¶
-
thoth.adviser.utils module¶
Various helpers and utility functions.
-
thoth.adviser.utils.
log_once
(logger: logging.Logger, log_state: Set[Any], log_state_key: Any, msg: str, *args: Any, level: int = 30, **kwargs: Any) → None[source]¶ Log the given message once.
-
thoth.adviser.utils.
should_keep_history
(value: Any) → bool[source]¶ Check if the history should be kept.
Used as click’s converter. If not set explicitly during invocation, check environment variable to turn off history tracking. The default value of `value’ is None which triggers checks in environment variables.
thoth.adviser.wrap module¶
A base class for implementing wrap units.
-
class
thoth.adviser.wrap.
Wrap
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Wrap base class implementation.
Module contents¶
Thoth’s adviser for recommending Python stacks.
-
class
thoth.adviser.
Beam
(width: Optional[int] = None, *, keep_history=None, beam_history: List[Tuple[int, Optional[float]]] = NOTHING)[source]¶ Bases:
object
Beam implementation.
The implementation of beam respecting beam width: https://en.wikipedia.org/wiki/Beam_search
Beam is internally implemented on top of a min-heap queue to optimize inserts and respect beam width in O(log(N)).
The most frequent operation performed on the beam is not always the same - it depends on the beam width and number of states generated. If number of states generated is more than the width of the beam, its reasonable to optimize insertions into the beam with checks on beam width.
If number of states generated is smaller than the beam width, the beam could be optimized for removal of states.
As the first scenario is seen in a real-world deployment, the beam uses min-heapq to keep addition to the beam with beam_width checks in O(log(N)) and removals of the states in O(log(N)). To satisfy removals in O(log(N)), the beam maintains a dictionary mapping a state to its index in the beam.
-
add_state
(state: thoth.adviser.state.State) → None[source]¶ Add state to the internal state listing (do it in O(log(N)) time.
-
get
(idx: int) → thoth.adviser.state.State[source]¶ Get i-th element from the beam (constant time), keep it in the beam.
This method is suitable for random state indexing in the beam (like in case of adaptive simulated annealing). The actual index is not into a sorted array and has no special meaning assigned - beam under the hood uses min-heapq (as of now), but the index used is not guaranteed to point to a heap-like data structure.
-
get_last
() → Optional[thoth.adviser.state.State][source]¶ Get state that was added in the previous resolution round.
-
iter_states
() → List[thoth.adviser.state.State][source]¶ Iterate over states, do not respect their score in order of iteration.
-
iter_states_sorted
(reverse: bool = True) → Generator[thoth.adviser.state.State, None, None][source]¶ Iterate over sorted states.
-
keep_history
¶
-
new_iteration
() → None[source]¶ Called once a new iteration is done in resolver.
Used to keep track of beam history and to keep track of states added.
-
plot
() → matplotlib.figure.Figure[source]¶ Plot temperature history of adaptive simulated annealing.
-
pop
(idx: Optional[int] = None) → thoth.adviser.state.State[source]¶ Pop i-th element from the beam and remove it from the beam (this is actually toppop).
If index is not provided, pop the largest item kept in the beam.
-
property
size
¶ Get the current size of beam.
-
width
¶
-
-
class
thoth.adviser.
Boot
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Boot base class implementation.
-
class
thoth.adviser.
Context
(*, project: thoth.python.project.Project, graph: thoth.storages.graph.postgres.GraphDatabase, library_usage: Optional[Dict[str, Any]], limit: int, count: int, beam: thoth.adviser.beam.Beam, recommendation_type: Optional[thoth.adviser.enums.RecommendationType] = None, decision_type: Optional[thoth.adviser.enums.DecisionType] = None, package_versions: Dict[Tuple[str, str, str], thoth.python.package_version.PackageVersion] = NOTHING, dependencies: Dict[str, Dict[Tuple[str, str, str], Set[Tuple[str, str, str]]]] = NOTHING, dependents: Dict[str, Dict[Tuple[str, str, str], Set[Tuple[Tuple[str, str, str], Optional[str], Optional[str], Optional[str]]]]] = NOTHING, sources: Dict[str, thoth.python.source.Source] = NOTHING, iteration: int = 0, cli_parameters: Dict[str, Any] = NOTHING, stack_info: List[Dict[str, Any]] = NOTHING, accepted_final_states_count: int = 0, discarded_final_states_count: int = 0, accepted_states: List[Tuple[Tuple[float, int], thoth.adviser.state.State]] = NOTHING, accepted_states_counter: int = 0)[source]¶ Bases:
object
Context carried during adviser’s pipeline run.
It’s suitable to cache entries such as PackageVersion to optimize memory usage and optimize overhead needed - for example for parsing version strings (this is lazily pre-cached in PackageVersion).
-
accepted_final_states_count
¶
-
beam
¶
-
cli_parameters
¶
-
count
¶
-
decision_type
¶
-
dependencies
¶
-
dependents
¶
-
discarded_final_states_count
¶
-
get_package_version
(package_tuple: Tuple[str, str, str], *, graceful: bool = False) → Optional[thoth.python.package_version.PackageVersion][source]¶ Get the given package version registered to the context.
-
get_top_accepted_final_state
() → Optional[thoth.adviser.state.State][source]¶ Get the best accepted final state so far computed by the resolution pipeline.
-
graph
¶
-
is_dependency_monkey
() → bool[source]¶ Check if the current context refers to a dependency monkey run.
-
iter_accepted_final_states
() → Generator[thoth.adviser.state.State, None, None][source]¶ Get accepted final states by resolution pipeline, states are not sorted.
-
iter_accepted_final_states_sorted
(reverse: bool = True) → Generator[thoth.adviser.state.State, None, None][source]¶ Get accepted final states by resolution pipeline sorted by score and their precedence.
-
iteration
¶
-
library_usage
¶
-
limit
¶
-
package_versions
¶
-
project
¶
-
recommendation_type
¶
-
register_accepted_final_state
(state: thoth.adviser.state.State) → None[source]¶ Register an accepted state by the resolution pipeline.
-
register_package_tuple
(package_tuple: Tuple[str, str, str], *, develop: bool, dependent_tuple: Optional[Tuple[str, str, str]] = None, extras: Optional[List[str]] = None, os_name: Optional[str], os_version: Optional[str], python_version: Optional[str]) → thoth.python.package_version.PackageVersion[source]¶ Register the given package tuple to pipeline context and return its package version representative.
-
register_package_version
(package_version: thoth.python.package_version.PackageVersion) → bool[source]¶ Register the given package version to the context.
-
sources
¶
-
stack_info
¶
-
-
class
thoth.adviser.
DecisionType
[source]¶ Bases:
thoth.adviser.enums._ExtendedEnum
Type of decision used in Dependency Monkey for generating stacks.
-
ALL
= 1¶
-
RANDOM
= 2¶
-
-
class
thoth.adviser.
DependencyMonkey
(*, resolver: thoth.adviser.resolver.Resolver, stack_output: str = '-', context: Optional[Dict[Any, Any]] = NOTHING, dry_run: bool = False, decision_type: thoth.adviser.enums.DecisionType = <DecisionType.ALL: 1>)[source]¶ Bases:
object
Dependency monkey creates stacks based on the configuration using ASA.
-
property
beam
¶ Get beam instance used in the resolver.
-
context
¶
-
decision_type
¶
-
dry_run
¶
-
property
predictor
¶ Get predictor for the current dependency monkey configuration.
-
resolve
(*, with_devel: bool = True, user_stack_scoring: bool = False) → thoth.adviser.dm_report.DependencyMonkeyReport[source]¶ Perform simulated annealing and run dependency monkey on products.
-
resolver
¶
-
stack_output
¶
-
property
-
class
thoth.adviser.
DependencyMonkeyReport
(skipped: int = 0, responses: List[Dict[str, Union[Dict[str, Any], str]]] = NOTHING)[source]¶ Bases:
object
Report produced by a Dependency Monkey run.
-
add_response
(response: str, product: thoth.adviser.product.Product) → None[source]¶ Add a new response to response listing.
-
skipped
¶
-
-
class
thoth.adviser.
Ecosystem
[source]¶ Bases:
thoth.adviser.enums._ExtendedEnum
Ecosystem known to recommendation engine.
-
PYTHON
= 1¶
-
-
class
thoth.adviser.
PipelineBuilder
[source]¶ Bases:
object
Builder responsible for creating pipeline configuration from the project and its library usage.
-
classmethod
from_dict
(dict_: Dict[str, Any]) → thoth.adviser.pipeline_config.PipelineConfig[source]¶ Instantiate pipeline configuration based on dictionary supplied.
-
classmethod
get_adviser_pipeline_config
(*, recommendation_type: thoth.adviser.enums.RecommendationType, graph: thoth.storages.graph.postgres.GraphDatabase, project: thoth.python.project.Project, library_usage: Optional[Dict[str, Any]]) → thoth.adviser.pipeline_config.PipelineConfig[source]¶ Get adviser’s pipeline configuration.
-
classmethod
get_dependency_monkey_pipeline_config
(*, decision_type: thoth.adviser.enums.DecisionType, graph: thoth.storages.graph.postgres.GraphDatabase, project: thoth.python.project.Project, library_usage: Optional[Dict[str, Any]]) → thoth.adviser.pipeline_config.PipelineConfig[source]¶ Get dependency-monkey’s pipeline configuration.
-
classmethod
-
class
thoth.adviser.
PipelineConfig
(boots: Dict[Optional[str], List[thoth.adviser.boot.Boot]] = NOTHING, pseudonyms: Dict[str, List[thoth.adviser.pseudonym.Pseudonym]] = NOTHING, sieves: Dict[Optional[str], List[thoth.adviser.sieve.Sieve]] = NOTHING, steps: Dict[Optional[str], List[thoth.adviser.step.Step]] = NOTHING, strides: Dict[Optional[str], List[thoth.adviser.stride.Stride]] = NOTHING, wraps: Dict[Optional[str], List[thoth.adviser.wrap.Wrap]] = NOTHING)[source]¶ Bases:
object
A configuration of a pipeline for dependency-monkey and for adviser.
-
property
boots
¶ Get all boots.
-
property
boots_dict
¶ Get boots as a dictionary mapping.
-
property
pseudonyms
¶ Get all pseudonyms.
-
property
pseudonyms_dict
¶ Get pseudonyms as a dictionary mapping.
-
property
sieves
¶ Get all sieves.
-
property
sieves_dict
¶ Get sieves as a dictionary mapping.
-
property
steps
¶ Get all steps.
-
property
steps_dict
¶ Get steps as a dictionary mapping.
-
property
strides
¶ Get all strides.
-
property
strides_dict
¶ Get strides as a dictionary mapping.
-
to_dict
() → Dict[str, List[Dict[str, Any]]][source]¶ Return this pipeline configuration in a dict representation.
-
property
wraps
¶ Get all wraps.
-
property
wraps_dict
¶ Get wraps as a dictionary mapping.
-
property
-
class
thoth.adviser.
Predictor
(*, keep_history=None)[source]¶ Bases:
object
A base class for implementing a predictor for resolver.
-
classmethod
assigned_context
(context: thoth.adviser.context.Context) → Generator[None, None, None][source]¶ Assign context to predictor.
-
property
context
¶ Get context in which the unit runs in.
-
finalize_state
(state_id: int) → None[source]¶ Finalizer called when the given state is about to be destructed by garbage collector.
Method suitable if predictor keeps internal state for states. Note that this method is not called for remaining states if the resolver terminates.
@param state_id: id of state that is about to be finalized
-
keep_history
¶
-
classmethod
obtain_default_configuration
(config_option_name: str) → Any[source]¶ Get the default value assigned to the given configuration option of a predictor instance.
-
post_run_report
(report: thoth.adviser.report.Report) → None[source]¶ Post-run method run after the resolving has finished - this method is called only if resolving with a report.
The default operation is a noop, but a predictor can perform any post-run operations in this method. This method should not raise any exception.
-
pre_run
() → None[source]¶ Pre-initialize the predictor.
This method is called before any resolving with a freshly instantiated context. The default operation is a noop, but predictor can perform any initial setup in this method. This method should not raise any exception.
-
abstract
run
() → Tuple[thoth.adviser.state.State, Tuple[str, str, str]][source]¶ Run the main method used to run the predictor.
-
set_reward_signal
(state: thoth.adviser.state.State, package_tuple: Tuple[str, str, str], reward: float) → None[source]¶ Signalize the reward.
@param state: (child) state for which the reward signal is triggered @param package_tuple: Python package that was added to the state causing the reward @param reward: set to nan if the given state was not accepted a special value
of inf notifies about a new final state
-
classmethod
-
class
thoth.adviser.
Product
(*, project: thoth.python.project.Project, score: float, justification: List[Dict[str, str]], advised_runtime_environment: Optional[thoth.common.config.runtime_environment.RuntimeEnvironment] = None, advised_manifest_changes: List[List[Dict[str, Any]]] = NOTHING)[source]¶ Bases:
object
A representation of an advised stack.
-
advised_manifest_changes
¶
-
advised_runtime_environment
¶
-
classmethod
from_final_state
(*, context: thoth.adviser.context.Context, state: thoth.adviser.state.State) → thoth.adviser.product.Product[source]¶ Instantiate advised stack from final state produced by adviser’s pipeline.
-
justification
¶
-
project
¶
-
score
¶
-
-
class
thoth.adviser.
PythonRecommendationOutput
[source]¶ Bases:
thoth.adviser.enums._ExtendedEnum
Recommendation generation respecting constraints on package-version level observations.
-
PIPENV
= 1¶
-
REQUIREMENTS
= 2¶
-
-
class
thoth.adviser.
RecommendationType
[source]¶ Bases:
thoth.adviser.enums._ExtendedEnum
Recommendation generation respecting constraints on package-version level observations.
-
LATEST
= 3¶
-
PERFORMANCE
= 4¶
-
SECURITY
= 5¶
-
STABLE
= 1¶
-
TESTING
= 2¶
-
-
class
thoth.adviser.
Report
(*, count: int, pipeline: thoth.adviser.pipeline_config.PipelineConfig, resolver_iterations: int = 0, accepted_final_states_count: int = 0, discarded_final_states_count: int = 0, stack_info: Optional[List[Dict[str, Any]]] = None, heapq: List[Tuple[Tuple[float, int], thoth.adviser.product.Product]] = NOTHING, heapq_counter: int = 0)[source]¶ Bases:
object
A report stating output of an adviser run.
-
accepted_final_states_count
¶
-
add_product
(product: thoth.adviser.product.Product) → bool[source]¶ Add adviser pipeline product to report.
-
count
¶
-
discarded_final_states_count
¶
-
iter_products
() → Generator[thoth.adviser.product.Product, None, None][source]¶ Iterate over products stored in this report, respect their scores.
-
iter_products_sorted
(reverse: bool = True) → Generator[thoth.adviser.product.Product, None, None][source]¶ Iterate over products stored in this report, respect their scores.
-
pipeline
¶
-
resolver_iterations
¶
-
property
stack_info
¶ Retrieve stack info of this report.
-
-
class
thoth.adviser.
Resolver
(*, pipeline: thoth.adviser.pipeline_config.PipelineConfig, project: thoth.python.project.Project, library_usage, graph: thoth.storages.graph.postgres.GraphDatabase, predictor: thoth.adviser.predictor.Predictor, recommendation_type: Optional[thoth.adviser.enums.RecommendationType] = None, decision_type: Optional[thoth.adviser.enums.DecisionType] = None, limit: int = 10000, count: int = 3, beam_width=-1, limit_latest_versions=-1, cli_parameters: Dict[str, Any] = NOTHING, stop_resolving: bool = False, log_iteration: int = 7500, beam: Optional[thoth.adviser.beam.Beam] = None, solver: Optional[thoth.adviser.solver.PythonPackageGraphSolver] = None, context: Optional[thoth.adviser.context.Context] = None, log_unresolved: Set[Tuple[str, str, str]] = NOTHING, log_unsolved: Set[str] = NOTHING, log_sieved: Set[str] = NOTHING, log_step_skip_package: Set[Tuple[str, str, str]] = NOTHING, log_step_not_acceptable: Set[Tuple[str, str, str]] = NOTHING, log_no_intersected: Set[Tuple[Tuple[str, str, str], str]] = NOTHING)[source]¶ Bases:
object
Resolver for resolving software stacks using pipeline configuration and a predictor.
-
DEFAULT_BEAM_WIDTH
= -1¶
-
DEFAULT_COUNT
= 3¶
-
DEFAULT_LIMIT
= 10000¶
-
DEFAULT_LIMIT_LATEST_VERSIONS
= -1¶
-
property
beam
¶ Get beam for storing states.
-
beam_width
¶
-
cli_parameters
¶
-
property
context
¶ Retrieve context bound to the current resolver.
-
count
¶
-
decision_type
¶
-
classmethod
get_adviser_instance
(*, predictor: thoth.adviser.predictor.Predictor, beam_width: Optional[int] = None, count: int = 3, graph: Optional[thoth.storages.graph.postgres.GraphDatabase] = None, library_usage: Optional[Dict[str, Any]] = None, limit: int = 10000, limit_latest_versions: Optional[int] = -1, project: thoth.python.project.Project, recommendation_type: thoth.adviser.enums.RecommendationType, pipeline_config: Union[thoth.adviser.pipeline_config.PipelineConfig, Dict[str, Any], None] = None, cli_parameters: Optional[Dict[str, Any]] = None) → thoth.adviser.resolver.Resolver[source]¶ Get instance of resolver based on the project given to recommend software stacks.
-
classmethod
get_dependency_monkey_instance
(*, predictor: thoth.adviser.predictor.Predictor, beam_width: Optional[int] = None, count: int = 3, graph: Optional[thoth.storages.graph.postgres.GraphDatabase] = None, library_usage: Optional[Dict[str, Any]] = None, limit_latest_versions: Optional[int] = -1, project: thoth.python.project.Project, decision_type: thoth.adviser.enums.DecisionType, pipeline_config: Union[thoth.adviser.pipeline_config.PipelineConfig, Dict[str, Any], None] = None, cli_parameters: Optional[Dict[str, Any]] = None) → thoth.adviser.resolver.Resolver[source]¶ Get instance of resolver based on the project given to run dependency monkey.
-
graph
¶
-
library_usage
¶
-
limit
¶
-
limit_latest_versions
¶
-
log_iteration
¶
-
pipeline
¶
-
predictor
¶
-
project
¶
-
recommendation_type
¶
-
resolve
(*, with_devel: bool = True, user_stack_scoring: bool = True) → thoth.adviser.report.Report[source]¶ Resolve software stacks and return resolver report.
-
resolve_products
(*, with_devel: bool = True, user_stack_scoring: bool = True) → Generator[thoth.adviser.product.Product, None, None][source]¶ Resolve raw products as produced by this resolver pipeline.
-
property
solver
¶ Get solver instance - solver implemented on top of graph database.
-
stop_resolving
¶
-
-
class
thoth.adviser.
Sieve
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Sieve base class implementation.
-
class
thoth.adviser.
State
(score: float = 0.0, iteration: int = 0, unresolved_dependencies=NOTHING, resolved_dependencies=NOTHING, parent=None, *, advised_runtime_environment: Optional[thoth.common.config.runtime_environment.RuntimeEnvironment] = None, advised_manifest_changes: List[List[Dict[str, Any]]] = NOTHING, justification: List[Dict[str, str]] = NOTHING)[source]¶ Bases:
object
Implementation of an adviser’s state in state space.
-
add_justification
(justification: List[Dict[str, str]]) → None[source]¶ Add new entries to the justification field.
-
add_resolved_dependency
(package_tuple: Tuple[str, str, str]) → None[source]¶ Add a resolved dependency into the state.
-
add_unresolved_dependency
(package_tuple: Tuple[str, str, str]) → None[source]¶ Add unresolved dependency into the state.
-
advised_manifest_changes
¶
-
advised_runtime_environment
¶
-
clone
() → thoth.adviser.state.State[source]¶ Return a swallow copy of this state that can be used as a next state.
-
classmethod
from_direct_dependencies
(direct_dependencies: Dict[str, List[thoth.python.package_version.PackageVersion]]) → thoth.adviser.state.State[source]¶ Create an initial state out of direct dependencies.
-
get_first_unresolved_dependency
(dependency_name: Optional[str] = None) → Tuple[str, str, str][source]¶ Get a very first unresolved dependency tuple.
-
get_random_first_unresolved_dependency
(dependency_name: Optional[str] = None) → Tuple[str, str, str][source]¶ Get a very first unresolved dependency tuple.
-
get_random_unresolved_dependency
(dependency_name: Optional[str] = None, prefer_recent: bool = True) → Tuple[str, str, str][source]¶ Get a very first unresolved dependency tuple.
-
iter_resolved_dependencies
() → Generator[Tuple[str, str, str], None, None][source]¶ Iterate over resolved dependencies.
-
iter_unresolved_dependencies
() → Generator[Tuple[str, str, str], None, None][source]¶ Iterate over unresolved dependencies.
-
iteration
¶
-
justification
¶
-
mark_dependency_resolved
(package_tuple: Tuple[str, str, str]) → None[source]¶ Mark the given dependency as resolved in the current state.
-
property
parent
¶ Retrieve parent to this state.
If None the state is top level state or parent is no longer maintained. Note the return value of None depends on actual gc runs.
-
remove_unresolved_dependency
(package_tuple: Tuple[str, str, str]) → None[source]¶ Remove the given unresolved dependency from state.
-
remove_unresolved_dependency_subtree
(package_name: str) → None[source]¶ Remove the whole dependency sub-tree from the state.
-
resolved_dependencies
¶
-
score
¶
-
set_unresolved_dependencies
(dependencies: Dict[str, List[Tuple[str, str, str]]]) → None[source]¶ Set unresolved dependencies - any unresolved dependencies will be overwritten.
-
unresolved_dependencies
¶
-
-
class
thoth.adviser.
Step
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Step base class implementation.
Configuration option mutli_package_resolution states whether a step should be run if package is resolved multiple times for the same stack.
-
CONFIGURATION_DEFAULT
= {'multi_package_resolution': False, 'package_name': None}¶
-
CONFIGURATION_SCHEMA
= <Schema({'package_name': Any(<class 'str'>, None, msg=None), 'multi_package_resolution': <class 'bool'>}, extra=PREVENT_EXTRA, required=False) object>¶
-
SCORE_MAX
= 1.0¶
-
SCORE_MIN
= -1.0¶
-
-
class
thoth.adviser.
Stride
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Stride base class implementation.
-
class
thoth.adviser.
Unit
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
object
A base class for implementing pipeline units - strides and steps.
-
CONFIGURATION_DEFAULT
= {'package_name': None}¶
-
CONFIGURATION_SCHEMA
= <Schema({'package_name': Any(<class 'str'>, None, msg=None)}, extra=PREVENT_EXTRA, required=False) object>¶
-
classmethod
assigned_context
(context: thoth.adviser.context.Context) → Generator[None, None, None][source]¶ Assign context to all units.
-
classmethod
compute_expanded_configuration
(configuration_dict: Optional[Dict[str, Any]]) → Dict[str, Any][source]¶ Compute configuration as they would be computed based on unit configuration.
-
property
configuration
¶ Get configuration of instantiated pipeline unit.
-
property
context
¶ Get context in which the unit runs in.
-
classmethod
get_aicoe_configuration
(package_version: thoth.python.package_version.PackageVersion) → Optional[Dict[str, Any]][source]¶ Get AICoE specific configuration encoded in the AICoE index URL.
-
classmethod
is_aicoe_release
(package_version: thoth.python.package_version.PackageVersion) → bool[source]¶ Check if the given package-version is AICoE release.
-
property
name
¶ Get name of this pipeline unit.
-
post_run
() → None[source]¶ Called after the resolution is finished.
This method should not raise any exception.
-
post_run_report
(report)[source]¶ Post-run method run after the resolving has finished - this method is called only if resolving with a report.
This method should not raise any exception.
-
pre_run
() → None[source]¶ Called before running any pipeline unit with context already assigned.
This method should not raise any exception.
-
classmethod
should_include
(builder_context: PipelineBuilderContext) → Optional[Dict[str, Any]][source]¶ Check if the given pipeline unit should be included in the given pipeline configuration.
-
unit_run
¶
-
-
class
thoth.adviser.
Wrap
(*, configuration: Dict[str, Any] = NOTHING, name: str = None, unit_run: bool = False)[source]¶ Bases:
thoth.adviser.unit.Unit
Wrap base class implementation.