Integrating with Thoth

Project Thoth can guide you on your software stacks. To consume Thoth’s recommendations, there are multiple ways on how to integrate:

  • Command line interface - Thamos CLI

  • Kebechet (GitHub application)

  • Jupyter Notebooks

  • OpenShift s2i build process

  • Thamos library (not fully supported yet as API might change)

Pre-requirements for your project

To let Thoth manage your Python application, your application has to use Pipenv to manage virtual environment and application dependencies. Pipenv’s two main files, Pipfile and Pipfile.lock, has to be placed in the root of your Python application directory.

If you use requirements.txt, the easiest way how to convert to Pipenv’s Pipfile and Pipfile.lock is to run the following command:

pipenv install --requirements requirements.txt

And add both file produced, Pipfile and Pipfile.lock, into your Git repository.

It’s also possible to use pip format as well as format used by pip-tools. To use these formats, you will need to adjust requirements_format configuration option in your .thoth.yaml configuration file.

Note

It’s recommended to use Pipenv if possible. Pipenv introduces more consistent files that track Python package indexes used as well as artifact hashes in the lock file explicitly.

By switching to pip/pip-compile file format the behaviour of file lookup is following (sorted based on priority):

  • if requirements.txt and requirements.in files are present, requirements.txt file is used as a lockfile and requirements.in states direct dependencies (pip-tools behavior)

  • if just requirements.in file is present, it is used as a file containing direct dependencies (pip-tools behaviour)

  • if just requirements.txt file is present, it is used as a file containing direct dependencies (raw pip behaviour)

Thoth’s output of requirements.txt uses implicitly hashes of artifacts - equivalent to pip-compile --generate-hashes. It is required to state artifact hashes if requirements.txt is treated as a lockfile.

Command Line Interface - Thamos CLI

The easiest way how to get recommendations from Thoth service is to install Thamos (Thoth’s CLI and library):

pip3 install -U thamos

And configure your project to use Thoth’s recommendations and ask for them:

cd your/project/path
thamos config
thamos advise

Before you ask for an advise, make sure your Pipenv’s files Pipfile and optionally Pipfile.lock are present in the root directory of your project:

ls -la your/project/path
..
.thoth.yaml
..
Pipfile
Pipfile.lock
..

Once Thoth responds back with recommendations, you can install your dependencies using:

pipenv install --deploy --dev

Please follow Thamos documentation for more info. Also check thoth-station/cli-examples repository that demonstrates an example application that using Thamos CLI.

OpenShift Python s2i build process

Thoth can be used in OpenShift’s s2i process where it can produce recommendations targeting your specific hardware configuration you use to run your application inside the cluster (e.g. specific GPU available in the cluster).

Note

Check thoth-station/s2i-example-migration with a step-by-step tutorial on how to port an already existing Python s2i application to Thoth.

You can find a list of base images which you can use with Thoth in s2i-thoth repository with detailed instructions on how to use Thoth in the OpenShift’s s2i process. The container images are hosted at quay.io/organization/thoth-station.

Note

You can use a tool called thoth-s2i that can automatically migrate your existing s2i application to use Thoth.

Thoth’s s2i container images can be configured using environment variables supplied to the build config:

  • THOTH_ADVISE - always use the recommended stack by Thoth (even if the lock file is present in the repo)

  • THOTH_PROVENANCE_CHECK - verify stack provenance - the provenance check is triggered only if the lock file is not comming from Thoth’s recommendation engine (otherwise the stack has already verified provenance)

  • THOTH_ASSEMBLE_DEBUG - run s2i’s assemble script in verbose mode

  • THOTH_DRY_RUN - submit stack to Thoth’s recommendation engine but do not use the recommended lock file, use the lock file present in the repo instead

  • THOTH_FROM_MASTER - Use Thamos from git instead of a PyPI release - handy if the released Thamos has a bug which was fixed in the master branch

  • THOTH_HOST - Thoth’s host to reach out to for recommendations (defaults to prod deployment at khemenu.thoth-station.ninja)

  • THOTH_ERROR_FALLBACK - fallback to the lock file present in the repository if the submitted Thoth analysis fails

See also configuration options for Thoth’s client present in Thamos documentation.

An example of such application can be found on GitHub - s2i TensorFlow example.

Kebechet - GitHub application

Here is are few easy steps describing how you can add Kebechet to your GitHub project:

  • Install Thamos CLI tool:

    pip3 install thamos  # keep in mind: requires Python 3.6+!!
    
  • Go to the repository that should be managed by Thoth which already has Pipfile present:

    cd ~/git/repo/
    
  • Setup Thamos configuration:

    thamos config
    

That’s it - thamos would create a “.thoth.yaml” file for you. Now you can change the managers you want to run. Here, for example, we want version and update manager to run on the demo repo so the configuration is:

managers:
    - name: update
      configuration:
        labels: [bot, kebechet]
    - name: version
      configuration:
        labels: [bot, kebechet]
        changelog_file: true

In case thamos config doesn’t work for you, add the .thoth.yaml file in your project root directory manually. An example can be found here.

Kebechet cares about the managers you add under the manager section. You will find how to define the manager config under each of the manager readme - kebechet/managers.

To see available runtime environment configurations, check Thoth endpoints. For available configurations issue:

curl -X GET "https://khemenu.thoth-station.ninja/api/v1/runtime-environment" -H  "accept: application/json"

Now that you are done with the setup of which managers you want to be run on your project, you are done with the major part. We would next install the GitHub app to ensure we receive webhooks from the repository, please install Khebhut, which is an alias for Kebechet.

And finally, let’s add our bot friend as a collaborator, so that he can help you maintain your project. In GitHub go to Settings -> Manage access and invite @sesheta as a collaborator.

Jupyter Notebooks

TODO: write a summary