Contents¶
Overview¶
| docs | |
|---|---|
| tests |  |
| package |      |
Checkon is a tool to help library maintainers ensure backward-compatibilty by running downstream applications’ test suites with library pre-release versions.
Supported meta-runners:
Supported test-runners:
Currently missing support for:
Installation¶
pip install checkon
Documentation¶
Usage¶
I maintain a library lib1. lib1 is used as a dependency by other libraries,
including lib2. I want to make some changes to lib1, but before releasing, I want
to make sure it won’t break the code of my users in lib2.
So I want to run lib2’s test suite on the new version of lib1.
$ checkon test \
--upstream-new ../lib1 \
--upstream-base git+https://github.com/metatooling/lib1.git@master \
dependents https://github.com/metatooling/lib2.git
Checkon will clone lib2, run its test suite via tox, and show if there are any
failures in the version on my branch specified by --upstream-new that pass under the
master version on GitHub specified by --upstream-base. (Expand table.)
envname application classname name line provider message text
--------- ----------------------------------- --------------- ---------- ------ ------------------------------------------- -------------------------------------------------------------- --------------------------------------------------------------------------------
py37 https://github.com/metatooling/lib2 tests.test_lib2 test_three 7 git+https://github.com/metatooling/lib1.git TypeError: add() takes 2 positional arguments but 3 were given def test_three():
> assert lib2.app.add_args([1, 2, 3]) == 6
tests/test_lib2.py:9:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = [1, 2, 3]
def add_args(args: t.List[int]) -> int:
> return lib1.app.add(*args)
E TypeError: add() takes 2 positional arguments but 3 were given
src/lib2/app.py:7: TypeError
Suppose I’m contributing to a popular project like attrs. I can retrieve a list of projects depending on it from the web:
$ checkon list dependents-from-librariesio --limit=5 attrs
https://github.com/pytest-dev/pytest
https://github.com/Julian/jsonschema
https://github.com/twisted/twisted
https://github.com/HypothesisWorks/hypothesis
https://github.com/pypa/packaging
And I can run all their tests using my forked version of attrs.
$ checkon test \
--upstream-new ../attrs \
--upstream-base git+https://github.com/python-attrs/attrs \
dependents-from-librariesio --limit=5 attrs
Or pick test suites in a configuration file. The file can specify repositories and tox environments to run.
# dependents.toml
[[dependents]]
repository = "https://github.com/Julian/jsonschema"
toxenv_regex = "py37"
[[dependents]]
repository = "https://github.com/twisted/twisted"
toxenv_regex = "py37"
$ checkon test \
--upstream-new ../attrs \
--upstream-base git+https://github.com/python-attrs/attrs \
dependents-from-file ./dependents.toml
I can check all the pull requests in the attrs repository against specified dependents.
$ checkon test \
--output-format=json \
--upstream-pull-requests https://github.com/python-attrs/attrs \
--upstream-base git+https://github.com/python-attrs/attrs@master \
dependents-from-file dependents.toml
Or check master against dependents, relative to the latest release.
$ checkon test \
--output-format=json \
--upstream-new https://github.com/python-attrs/attrs@master \
--upstream-base git+https://github.com/python-attrs/attrs@19.1.0 \
dependents-from-file dependents.toml
Reference¶
checkon package¶
Submodules¶
checkon.app module¶
checkon.cli module¶
checkon.results module¶
-
class
checkon.results.AppSuiteRun(upstreamed: str, dependent_result: checkon.results.DependentResult)[source]¶ Bases:
object
-
class
checkon.results.Comparison(base_requirement: str, new_requirement: str, base_failures: FrozenSet[checkon.results.FailedTest], new_failures: FrozenSet[checkon.results.FailedTest])[source]¶ Bases:
object
-
class
checkon.results.DependentResult(url: str, suite_runs: List[checkon.results.ToxTestSuiteRun])[source]¶ Bases:
object
-
class
checkon.results.FailedTest(name: str, classname: str, file: Optional[str], line: Optional[str], failure)[source]¶ Bases:
object
-
class
checkon.results.FailureField(*, default: Any = <marshmallow.missing>, missing: Any = <marshmallow.missing>, data_key: str = None, attribute: str = None, validate: Union[Callable[[Any], Any], Sequence[Callable[[Any], Any]], Generator[Callable[[Any], Any], None, None]] = None, required: bool = False, allow_none: bool = None, load_only: bool = False, dump_only: bool = False, error_messages: Dict[str, str] = None, **metadata)[source]¶ Bases:
marshmallow.fields.Field
-
class
checkon.results.TestCaseRun(name: str, classname: str, file: Union[str, NoneType], line: Union[int, NoneType], time: str, failure: Union[checkon.results.Failure, NoneType] = None, skipped: Any = None, system_err: Union[Any, NoneType] = None)[source]¶ Bases:
object-
failure= None¶
-
skipped= None¶
-
system_err= None¶
-
-
class
checkon.results.TestSuiteRun(errors: int, failures: int, tests: int, time: str, timestamp: Union[datetime.datetime, NoneType], hostname: Union[str, NoneType], name: Union[str, NoneType], test_cases: List[checkon.results.TestCaseRun], envname: Union[str, NoneType], skipped: Union[int, NoneType] = None)[source]¶ Bases:
object-
skipped= None¶
-
checkon.satests module¶
-
class
checkon.satests.Application(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
application_id¶
-
name¶
-
-
class
checkon.satests.Database(engine: Any, session: Any, cache: Dict[KT, VT] = NOTHING)[source]¶ Bases:
object
-
class
checkon.satests.FailureOutput(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
failure_output_id¶
-
message¶
-
text¶
-
-
class
checkon.satests.Provider(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
provider_id¶
-
requirement¶
-
-
class
checkon.satests.TestCase(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
classname¶
-
file¶
-
line¶
-
name¶
-
test_case_id¶
-
-
class
checkon.satests.TestCaseRun(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
duration¶
-
test_case¶
-
test_case_id¶
-
test_case_run_id¶
-
test_failure¶
-
test_failure_id¶
-
test_suite_run¶
-
test_suite_run_id¶
-
-
class
checkon.satests.TestFailure(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
failure_output¶
-
failure_output_id¶
-
test_failure_id¶
-
-
class
checkon.satests.TestSuite(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
test_cases¶
-
test_cases_id¶
-
test_suite_id¶
-
-
class
checkon.satests.TestSuiteRun(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
duration¶
-
envname¶
-
start_time¶
-
test_case_runs¶
-
test_suite_run_id¶
-
-
class
checkon.satests.ToxRun(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
application¶
-
provider¶
-
tox_run_id¶
-
toxenv_runs¶
-
-
class
checkon.satests.Toxenv(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
application¶
-
application_id¶
-
name¶
-
toxenv_id¶
-
-
class
checkon.satests.ToxenvRun(**kwargs)[source]¶ Bases:
sqlalchemy.ext.declarative.api.Base-
envname¶
-
start_time¶
-
test_suite_run¶
-
test_suite_run_id¶
-
tox_run¶
-
tox_run_id¶
-
toxenv¶
-
toxenv_id¶
-
toxenv_run_id¶
-
checkon.tests module¶
-
class
checkon.tests.ProviderApplicationToxEnvRun(provider: checkon.tests.Provider, application: checkon.tests.Application, toxenv_run: checkon.tests.ToxEnvRun)[source]¶ Bases:
object
-
class
checkon.tests.TestCase(name: str, classname: str, file: str, line: int, skipped: Any)[source]¶ Bases:
object
-
class
checkon.tests.TestCaseRun(duration: str, test_case: checkon.tests.TestCase)[source]¶ Bases:
object
-
class
checkon.tests.TestFailure(output: checkon.tests.FailureOutput, test_case_run: checkon.tests.TestCaseRun)[source]¶ Bases:
object
-
class
checkon.tests.TestSuiteRun(test_suite: checkon.tests.TestSuite, start_time: datetime.datetime, duration: Any)[source]¶ Bases:
object
-
class
checkon.tests.ToxEnv(name: str, application: checkon.tests.Application)[source]¶ Bases:
object
checkon.tox module¶
-
class
checkon.tox.Python(is_64: bool, version_info: Tuple[int, int, int, str, int], executable: str, name: str, sysplatform: str, version: str)[source]¶ Bases:
object
-
class
checkon.tox.TestEnv(test: List[checkon.tox.Test], installed_packages, python: checkon.tox.Python, setup: List[checkon.tox.Setup], name: Optional[str])[source]¶ Bases:
object
Module contents¶
Contributing¶
Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
Bug reports¶
When reporting a bug please include:
- Your operating system name and version.
- Any details about your local setup that might be helpful in troubleshooting.
- Detailed steps to reproduce the bug.
Documentation improvements¶
checkon could always use more documentation, whether as part of the official checkon docs, in docstrings, or even on the web in blog posts, articles, and such.
Feature requests and feedback¶
The best way to send feedback is to file an issue at https://github.com/metatooling/checkon/issues.
If you are proposing a feature:
- Explain in detail how it would work.
- Keep the scope as narrow as possible, to make it easier to implement.
- Remember that this is a volunteer-driven project, and that code contributions are welcome :)
Development¶
To set up checkon for local development:
Fork checkon (look for the “Fork” button).
Clone your fork locally:
git clone git@github.com:your_name_here/checkon.git
Create a branch for local development:
git checkout -b name-of-your-bugfix-or-feature
Now you can make your changes locally.
When you’re done making changes, run all the checks, doc builder and spell checker with tox one command:
toxCommit your changes and push your branch to GitHub:
git add . git commit -m "Your detailed description of your changes." git push origin name-of-your-bugfix-or-feature
Submit a pull request through the GitHub website.
Pull Request Guidelines¶
If you need some code review or feedback while you’re developing the code just make the pull request.
For merging, you should:
- Include passing tests (run
tox) [1]. - Update documentation when there’s new API, functionality etc.
- Add a file in
changelog.d/describing the changes. The filename should be{id}.{type}.rst, where{id}is the number of the GitHub issue or pull request and{type}is one ofbreaking(for breaking changes),deprecation(for deprecations), orchange(for non-breaking changes). For example, to add a new feature requested in GitHub issue #1234, add a file calledchangelog.d/1234.change.rstdescribing the change. - Add yourself to
AUTHORS.rst.
| [1] | If you don’t have all the necessary python versions available locally you can rely on Travis - it will run the tests for each change you add in the pull request. It will be slower though … |
Tips¶
To run a subset of tests:
tox -e envname -- pytest -k test_myfeature
To run all the test environments in parallel (you need to pip install detox):
detox
Authors¶
- Checkon contributors - https://github.com/metatooling/checkon