Development

This page describes the development of sdmx. Contributions are welcome!

• For current development priorities, see the list of GitHub milestones and issues/PRs targeted to each.

• For wishlist features, see issues on GitHub tagged ‘enh’ or ‘wishlist’.

Code style

• This project uses, via pre-commit:

  • ruff for code style and linting, including:

    • ensure PEP 8 compliance, and

    • ensure a consistent order for imports (superseding flake8 and isort).

  • mypy for static type checking.

  These must be applied to new or modified code. This can be done manually, or through code editor plug-ins. Pre-commit hooks for git can be installed via:

  pip install pre-commit
  pre-commit install
  

  These will ensure that each commit is compliant with the code style.

• The pytest.yaml GitHub Actions workflow checks code quality for pull requests and commits. This check must pass for pull requests to be merged.

• Follow the 7 rules of a great Git commit message.

• Write docstrings in the numpydoc style.

Test specimens

New in version 2.0.

A variety of specimens—example files from real web services, or published with the standards—are used to test that sdmx correctly reads and writes the different SDMX message formats. Since v2.0, specimens are stored in the separate sdmx-test-data repository.

Running the test suite requires these files. To retrieve them, use one of the following methods:

1. Obtain the files by one of two methods:

   1. Clone khaeru/sdmx-test-data:

      $ git clone git@github.com:khaeru/sdmx-test-data.git
      

   2. Download khaeru/sdmx-test-data

2. Indicate where pytest can find the files, by one of two methods:

   1. Set the SDMX_TEST_DATA environment variable:

      # Set the variable only for one command
      $ SDMX_TEST_DATA=/path/to/files pytest
      
      # Export the variable to the environment
      $ export SDMX_TEST_DATA
      $ pytest
      

   2. Give the option --sdmx-test-data=<PATH> when invoking pytest:

      $ pytest --sdmx-test-data=/path/to/files
      

The files are:

• Arranged in directories with names matching particular sources in sources.json.

• Named with:

  • Certain keywords:

    • -structure: a structure message, often associated with a file with a similar name containing a data message.

    • ts: time-series data, i.e. with a TimeDimensions at the level of individual Observations.

    • xs: cross-sectional data arranged in other ways.

    • flat: flat DataSets with all Dimensions at the Observation level.

    • ss: structure-specific data messages.

  • In some cases, the query string or data flow/structure ID as the file name.

  • Hyphens - instead of underscores _.

Releasing

Before releasing, check:

• khaeru/sdmx to ensure that the push and scheduled builds are passing.

• https://readthedocs.org/projects/sdmx1/builds/ to ensure that the docs build is passing.

Address any failures before releasing.

1. Create a new branch:

   $ git checkout -v release/X.Y.Z
   

2. Edit doc/whatsnew.rst. Comment the heading “Next release”, then insert another heading below it, at the same level, with the version number and date.

3. Make a commit with a message like “Mark vX.Y.Z in doc/whatsnew”.

4. Tag the version as a release candidate, i.e. with a rcN suffix, and push:

   $ git tag vX.Y.ZrcN
   $ git push --tags --set-upstream origin release/X.Y.Z
   

5. Open a pull request with the title “Release vX.Y.Z” using this branch. Check:

   • at khaeru/sdmx that the workflow completes: the package builds successfully and is published to TestPyPI.

   • at https://test.pypi.org/project/sdmx1/ that:

     • The package can be downloaded, installed and run.

     • The README is rendered correctly.

   If needed, address any warnings or errors that appear and then continue from step (3), i.e. make (a) new commit(s) and tag, incrementing the release candidate number, e.g. from rc1 to rc2.

6. Merge the PR using the “rebase and merge” method.

7. (optional) Tag the release itself and push:

   $ git tag vX.Y.Z
   $ git push --tags origin main
   

   This step (but not step (3)) can also be performed directly on GitHub; see (7), next.

8. Visit khaeru/sdmx and mark the new release: either using the pushed tag from (7), or by creating the tag and release simultaneously.

9. Check at khaeru/sdmx and https://pypi.org/project/sdmx1/ that the distributions are published.

Internal code reference

testing: Testing utilities

class sdmx.testing.MessageTest

Bases: object

Base class for tests of specific specimen files.

directory: str | Path = PosixPath('.')

filename: str

msg(path)

path(test_data_path)

class sdmx.testing.SpecimenCollection(base_path)

Bases: object

Collection of test specimens.

as_params(format=None, kind=None, marks={})

Generate pytest.param() from specimens.

One param() is generated for each specimen that matches the format and kind arguments (if any). Marks are attached to each param from marks, wherein the keys are partial paths.

expected_data(path)

Return the expected to_pandas() result for the specimen path.

specimens: List[Tuple[Path, str, str]]

sdmx.testing.XFAIL = {'unsupported': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'strict': True, 'reason': 'Not implemented by service', 'raises': (<class 'requests.exceptions.HTTPError'>, <class 'NotImplementedError'>, <class 'ValueError'>)})), 503: MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'raises': <class 'requests.exceptions.HTTPError'>, 'reason': '503 Server Error: Service Unavailable'}))}

Marks for use below.

sdmx.testing.assert_pd_equal(left, right, **kwargs)

Assert equality of two pandas objects.

sdmx.testing.generate_endpoint_tests(metafunc)

pytest hook for parametrizing tests that need an “endpoint” fixture.

This function relies on the DataSourceTest base class defined in test_sources. It:

• Generates one parametrization for every Resource (= REST API endpoint).

• Applies pytest “xfail” (expected failure) marks according to:

  1. Source.supports, i.e. if the particular source is marked as not supporting certain endpoints, the test is expected to fail.

  2. DataSourceTest.xfail, any other failures defined on the source test class (e.g. DataSourceTest subclass).

  3. DataSourceTest.xfail_common, common failures.

sdmx.testing.mock_service_adapter()

sdmx.testing.parametrize_specimens(metafunc)

Handle @pytest.mark.parametrize_specimens(…).

sdmx.testing.pytest_addoption(parser)

Add the --sdmx-test-data command-line option to pytest.

sdmx.testing.pytest_configure(config)

Handle the --sdmx-test-data command-line option.

sdmx.testing.pytest_generate_tests(metafunc)

Generate tests.

Calls both parametrize_specimens() and generate_endpoint_tests().

sdmx.testing.specimen(pytestconfig)

Fixture: the SpecimenCollection.

sdmx.testing.test_data_path(pytestconfig)

Fixture: the Path given as –sdmx-test-data.

sdmx.testing.testsource()

Fixture: the Source.id of a non-existent data source.

util: Utilities

Inline TODOs

Todo

Support selection of language for conversion of InternationalString.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/sdmx1/checkouts/latest/doc/api/writer.rst, line 80.)

Todo

Currently other functions in writer.xml all pass the style argument to this function. As an enhancement, allow user or automatic selection of different reference styles.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/sdmx1/envs/latest/lib/python3.12/site-packages/sdmx/writer/xml.py:docstring of sdmx.writer.xml.reference, line 3.)