Development¶
This page describes the development of sdmx
.
Contributions are welcome!
For current development priorities, see the list of GitHub milestones and issues/PRs targeted to each.
For wishlist features, see issues on GitHub tagged ‘enh’ or ‘wishlist’.
Code style¶
Apply the following to new or modified code:
isort -rc . && black . && mypy . && flake8
Respectively, these:
Write docstrings in the numpydoc style.
Test specimens¶
New in version 2.0.
A variety of specimens—example files from real web services, or published with the standards—are used to test that sdmx
correctly reads and writes the different SDMX message formats.
Since v2.0, specimens are stored in the separate sdmx-test-data repository.
Running the test suite requires these files. To retrieve them, use one of the following methods:
Obtain the files by one of two methods:
Clone
khaeru/sdmx-test-data
:$ git clone git@github.com:khaeru/sdmx-test-data.git
Download https://github.com/khaeru/sdmx-test-data/archive/master.zip
Indicate where pytest can find the files, by one of two methods:
Set the SDMX_TEST_DATA environment variable:
# Set the variable only for one command $ SDMX_TEST_DATA=/path/to/files pytest # Export the variable to the environment $ export SDMX_TEST_DATA $ pytest
Give the option
--sdmx-test-data=<PATH>
when invoking pytest:$ pytest --sdmx-test-data=/path/to/files
The files are:
Arranged in directories with names matching particular sources in
sources.json
.Named with:
Certain keywords:
-structure
: a structure message, often associated with a file with a similar name containing a data message.ts
: time-series data, i.e. with a TimeDimensions at the level of individual Observations.xs
: cross-sectional data arranged in other ways.flat
: flat DataSets with all Dimensions at the Observation level.ss
: structure-specific data messages.
In some cases, the query string or data flow/structure ID as the file name.
Hyphens
-
instead of underscores_
.
Releasing¶
Before releasing, check:
https://github.com/khaeru/sdmx/actions?query=workflow:pytest+branch:master to ensure that the push and scheduled builds are passing.
https://readthedocs.org/projects/sdmx1/builds/ to ensure that the docs build is passing.
Address any failures before releasing.
Edit
doc/whatsnew.rst
to replace “Next release” with the version number and date. Make a commit with a message like “Mark vX.Y.Z in whatsnew.rst”.Tag the version as a release candidate, e.g.:
$ git tag v1.2.3rc1
Test-build and check the source and binary packages:
$ rm -rf build dist $ python setup.py bdist_wheel sdist $ twine check dist/*
Address any warnings or errors that appear. If needed, make a new commit and go back to step (2).
Upload the packages to the PyPI test repository:
$ twine upload -r testpypi dist/*
Check at https://test.pypi.org/project/sdmx1/ that:
The package can be downloaded, installed and run.
The README is rendered correctly.
Links to the documentation go to the correct version.
If not, modify the code and go back to step (2).
Tag the release:
$ git tag v1.2.3
If this is the same commit as the release candidate tag, delete that tag.
Built and upload to both PyPI and the test repo (to supersede the RC):
$ rm -rf build dist $ python setup.py bdist_wheel sdist $ twine check dist/* $ twine upload dist/* $ twine upload -r testpypi dist/*
Edit
doc/whatsnew.rst
to add a new heading for the next release. Make a commit with a message like “Reset whatsnew.rst to development state”.Push the commits and tag to GitHub:
$ git push --tags
Visit https://github.com/khaeru/sdmx/releases and mark the new release using the pushed tag.
Internal code reference¶
testing
: Testing utilities¶
-
class
sdmx.testing.
MessageTest
[source]¶ Bases:
object
Base class for tests of specific specimen files.
-
directory
: Union[str, pathlib.Path] = PosixPath('.')¶
-
-
class
sdmx.testing.
SpecimenCollection
(base_path)[source]¶ Bases:
object
Collection of test specimens.
-
as_params
(format=None, kind=None, marks={})[source]¶ Generate
pytest.param()
from specimens.One
param()
is generated for each specimen that matches the format and kind arguments (if any). Marks are attached to each param from marks, wherein the keys are partial paths.
-
expected_data
(path)[source]¶ Return the expected
to_pandas()
result for the specimen path.
-
-
sdmx.testing.
generate_endpoint_tests
(metafunc)[source]¶ pytest hook for parametrizing tests that need an “endpoint” fixture.
-
sdmx.testing.
pytest_addoption
(parser)[source]¶ Add the
--sdmx-test-data
command-line option to pytest.
-
sdmx.testing.
pytest_generate_tests
(metafunc)[source]¶ Generate tests.
Calls both
parametrize_specimens()
andgenerate_endpoint_tests()
.
-
sdmx.testing.
specimen
(pytestconfig)[source]¶ Fixture: the
SpecimenCollection
.
-
sdmx.testing.
unsupported
= MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'strict': True, 'reason': 'Known non-supported endpoint.', 'raises': <class 'NotImplementedError'>}))¶ This exception is raised by client.Client._request_from_args
Todo
parametrize force=True to query these endpoints anyway; then XPASS will reveal when data sources change their support for endpoints
util
: Utilities¶
-
class
sdmx.util.
BaseModel
[source] Bases:
pydantic.main.BaseModel
Shim for pydantic.BaseModel.
This class changes two behaviours in pydantic. The methods are direct copies from pydantic’s code, with marked changes.
https://github.com/samuelcolvin/pydantic/issues/524
“Multiple RecursionErrors with self-referencing models”
In e.g.
Item
, having both .parent and .child references leads to infinite recursion during validation.Fix: override BaseModel.__setattr__.
New value ‘limited’ for Config.validate_assignment: no sibling field values are passed to Field.validate().
New key Config.validate_assignment_exclude: list of field names that are not validated per se and not passed to Field.validate() when validating a sibling field.
https://github.com/samuelcolvin/pydantic/issues/521
“Assignment to attribute changes id() but not referenced object,” marked as wontfix by pydantic maintainer.
When cls.attr is typed as BaseModel (or a subclass), then a.attr is b.attr is always False, even when set to the same reference.
Fix: override BaseModel.validate() without copy().
-
class
Config
[source] Bases:
object
-
validate_assignment
= 'limited'
-
validate_assignment_exclude
: List[str] = []
-
-
classmethod
validate
(value: Any) → Model[source]
-
sdmx.util.
summarize_dictlike
(dl, maxwidth=72)[source] Return a string summary of the DictLike contents.
-
sdmx.util.
validate_dictlike
(*fields)[source]
Inline TODOs¶
Todo
Support selection of language for conversion of
InternationalString
.
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/sdmx1/checkouts/v2.0.0/doc/api.rst, line 139.)
Todo
parametrize force=True to query these endpoints anyway; then XPASS will reveal when data sources change their support for endpoints
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/sdmx1/envs/v2.0.0/lib/python3.7/site-packages/sdmx/testing.py:docstring of sdmx.testing.unsupported, line 3.)