What’s new?¶
v2.26.0 (2026-04-04)¶
Add OECD3 source (PR #278, thanks @dosse for #272), update documentation URLs.
Read dataset-related attributes from SDMX-ML 2.1 (PR #276, thanks @aboddie: for #266, PR #267).
Read
<structure:MetadataAttributeUsage>from SDMX-ML 3.1 (#271, PR #275).New class
MetadataAttributeUsage; see its docstring for implementation notes.
Use “*” as the default provider and resource ID in SDMX-REST 2.x (SDMX 3) URLs (PR #278).
Add override keyword arguments to
urn.make()(PR #275).Bug fix for reading SDMX-ML 3.x where certain elements (for instance,
<structure:Code>) had no internal whitespace whatsoever (PR #278).
v2.25.1 (2026-01-23)¶
sdmxis tested and compatible with pandas version 3.0.0, released 2026-01-21 (PR #268).sdmxis compatible with, and requires at minimum packaging version 26.0, released 2026-01-20 (PR #268).
v2.25.0 (2025-12-24)¶
Improve semantics and behaviour of
Version.increment()(PR #264). Previously,Version("1.2.3-dev4").increment(minor=1)would give “1.3.3-dev1”; now, the parts ‘inferior’ to the incremented part(s) are zeroed by default: “1.3.0”.
v2.24.0 (2025-12-18)¶
Python 3.14 (released 2025-10-07) is fully supported (PR #249).
Python 3.9 support is dropped, as it has reached end-of-life (PR #249).
sdmxrequires Python 3.10 or later.URNparses letters in the version part of a URN (#230, PR #252). This fixes a bug in v2.16.0–v2.23.1 where creatingVersionableArtefactwith bothversion=...andurn=...would raiseValueErroreven if the two were in agreement.Fix two regressions in
to_pandas()introduced in v2.23.0 (#251, PR #252).Fix false cache hits and misses (thanks @benfrankel for #256, PR #257).
Fix a bug where supplying references=… to ESTAT or EMPL would raise
ValueError(thanks @benfrankel for #259, PR #260).Adjust
Client.get()to avoid logging or returning a message from cache when dry_run=True (thanks @benfrankel for PR #261).
v2.23.1 (2025-10-01)¶
Bug fix for
convert.pandas/writer.csv(PR #247): in v2.23.0 only,PandasConverterraisedAttributeErrorwhen created.
v2.23.0 (2025-09-30)¶
Migration notes¶
to_pandas(..., rtype="compat")is deprecated and will be removed in a future release. Giving this argument raisesDeprecationWarning. User code should be adapted to explicitly mutate the objects returned byto_pandas().to_pandas(..., datetime=...)is deprecated and will be removed in a future release. Giving this argument raisesDeprecationWarning. User code should be adapted to explicitly pass values fordatetime_axis,datetime_dimension, and/ordatetime_freq, as required.
All changes¶
Expand
model,reader.xml, andwriter.xmlsupport for Registration and maintenance messages (PR #234). See the documentation for implementation details and errata in the standards documents.New classes
model.common.MessageText,StatusMessage,SubmissionResult, andSubmissionStatusType.New classes
message.RegistryInterfaceandSubmitStructureResponse.
New module
sdmx.comparethat collects logic for recursive comparison of SDMX artefacts (PR #234).New mix-in
Comparablethat adds acompare()method to subclasses.New class
compare.Optionsto control comparison behaviour and logging.sdmx.util.compare()is deprecated and will be removed in a future version.
to_csv()andto_pandas()supportLabels.bothandLabels.name(PR #243, PR #244, thanks @aboddie for PR #242).New modules (PR #243, PR #244):
convertandconvert.common.convert.pandasandPandasConverter, replacing.writer.pandas.format.commonand classesFormatandFormatOptions.format.csv.common,v1, andv2.typesfor type hinting first-party and downstream code.
Improve
Key:Key is sortable (PR #234).
Key.copy()returns the same type for subclasses (PR #243).
DataStructure.make_keyassociatesCodetoKeyValue.valuewhenRepresentation.enumeratedis set for the respectiveDimension.local_representation(PR #244).install_schemas()andconstruct_schema()fetch, store, and use a local copy ofxhtml1-strict.dsd(PR #236, #235). This enables use ofvalidate_xml()with lxml version 6.0.0 (released 2025-06-26) for SDMX-ML messages containing XHTML values.Correct a broken link to Information model (SDMX-IM) in the README (PR #233; thanks @econometricsfanboy for #232).
Update the base URL of the ILO source to use HTTPS instead of plain HTTP (PR #237).
New utilities
CompareTestsandpreserve_dunders()(PR #234);dimensions_to_attributes()(PR #243).Documentation for Standard formats moved to its own page (PR #243).
v2.22.0 (2025-03-25)¶
Migration notes¶
Modify code that imports
Annotationfromsdmx.model.commonto import from eithersdmx.model.v21orsdmx.model.v30, as appropriate. For example, instead of:from sdmx.model.common import Annotation a = Annotation(id="FOO", ...)
…do:
from sdmx.model.v21 import Annotation a = Annotation(id="FOO", ...)
Adjust code that accesses
ReportStructurevia thev21.MetadataSet.described_byattribute:To access ReportStructure, use the new
report_structureattribute.To access
MetadataStructureDefinition, usedescribed_by.
All changes¶
StructureMessage.get()handles full and partialURNs(PR #227).v21.Annotationandv30.Annotationare derived fromcommon.BaseAnnotation(PR #227). This allows to reflect that the latter has an attribute,v30.Annotation.value, that the former does not. This is a change in the SDMX 3.0.0 Information Model that is not mentioned in the “Summary of major changes and new functionality” or IM document.Code like
from sdmx.model.common import Annotationnow emitsDeprecationWarning, and in the future will raiseImportError.validate_xml()now supports<com:StructuredText>elements representing, for instance,XHTMLAttributeValue(PR #227). A new functionconstruct_schema()modifies the official SDMX-ML schemas to insert an import of the XML Schema for XHTML 1.0, allowing to validate the XHTML content within these elements.-
IdentifiableArtefactis comparable withstrvia itsid. This means thatsorted()can be used with mixed collections of these two types.Structure.groupingnow returns a list ofComponentList. Insdmxv2.21.1 and earlier, this list would include adictof 0 or moreGroupDimensionDescriptor, keyed by the ID of each. Now, each group dimension descriptor is directly an item in the list.v21.MetadataSet.report_structureis added and distinguished fromdescribed_by. This works around an issue in the SDMX 2.1 IM; see the class docstring for details.New convenience methods
MetadataReport.get(),MetadataReport.get_value(), andReportedAttribute.get_child().
Improve reading and writing of SDMX-ML (PR #227):
Read
<str:AnnotationValue>in SDMX-ML 3.0.0 (#226).Read
<str:Hierarchy>where the optional<... leveled="...">attribute is not present (#226).Read and write XSD-valid
v21.MetadataSetandv21.HierarchicalCodelist.Write
Dimension.concept_role.Write annotations associated with
DataSet,MetadataSet, andMetadataReport.Pending resolution of #228, ignore
<com:Link>in SDMX-ML 3.0.0 .
Rename IMF_beta, IMF_beta3 data sources to IMF_DATA, IMF_DATA3 and update documentation on 3 distinct IMF-run web services (thanks @aboddie for PR #225 and #224).
Update and expand Standards versions in the documentation (PR #227). A table is now included showing the correspondence of versions of component SDMX standards.
v2.21.1 (2025-01-14)¶
v2.21.0 (2025-01-13)¶
Add AR1, StatCan, and UY110 data sources (PR #218, #186, #187, #188).
Add IMF_beta, IMF_beta3 data sources and expand documentation on 3 distinct IMF-run web services (PR #218, #38).
New function
get_source()for case-insensitive lookup of sources (PR #218).Clientwill handle, for instance,Client("wb")the same asClient("WB")and log a message about the difference.Simplify
Sessionvia direct inheritance fromrequests_cache.session.CacheMixin, where installed (PR #217).Add an optional
session=...keyword argument toClient(PR #217).Add an optional
max_errors=...keyword argument tovalidate_xml()(PR #218).Improve network and offline tests via new and improved test utilities (PR #217). New test fixtures
session_with_pytest_cache()andsession_with_stored_responses().Tolerate invalid SDMX returned by BIS (and possibly other sources) that contains references to the non-existent
PublicationTableclass (PR #218, #38).Bug fix for reading
<str:Categorisation>from SDMX-ML 2.1: theCategorisation.categoryattribute was read as an instance of Categorisation, rather than Category (PR #215).Bug fix for reading
<mes:URI>and<mes:Telephone>from SDMX-ML 2.1<mes:Header>(PR #218). Up to v2.20.0, these causedNotImplementedError.
v2.20.0 (2024-12-16)¶
Add
to_sdmx()andDataFrameConverterto allow convertingpandas.DataFrameas if it were SDMX-CSV (PR #212).See also
Converter,CONVERTER,get_converter()for opportunities to extend this generic capability.Add
get_reader(); deprecatedetect_content_reader(),get_reader_for_media_type(),get_reader_for_path().Add
BaseReader.handles()andbinary_content_startswith; deprecatedetect(),supports_suffix(),handles_media_type().
Improve tolerance of invalid references in SDMX-ML (PR #207; thanks @nicolas-graves for #205). Where a file gives a reference for a
Component.concept_identity(such as for aDimensionorPrimaryMeasure) that is invalid—that is, the specifiedConceptdoes not exist in the referencedConceptScheme—log on levellogging.WARNINGand discard the reference. Previously such invalid references caused aKeyError. Prompted by an example in INSEE.Update the base URL of the WB source to use HTTPS instead of plain HTTP (PR #207).
Bug fix for writing
NameableArtefactto SDMX-ML (PR #211; thanks @3nz01 for #210). Up to v2.19.1, the<com:Annotations>element was written after elements such as<com:Name>, which is opposite the order given in the XSD schemas for SDMX-ML.sdmx.reader.xmltolerates non-standard element order, but some other implementations do not.
v2.19.1 (2024-10-23)¶
v2.19.0 (2024-10-23)¶
reader.csvsupports reading SDMX-CSV 2.0.0 (corresponding to SDMX 3.0.0) (PR #201, #34). See the implementation notes for information about the differences between the SDMX-CSV 1.0 and 2.0.0 formats and their support insdmx.Bug fix for writing
VersionableArtefactto SDMX-ML 2.1:KeyErrorwas raised ifVersionableArtefact.versionwas an instance ofVersion(PR #198).Bug fix for reading data from structure-specific SDMX-ML:
XMLParseError/NotImplementedErrorwas raised if reading 2 messages in sequence with different XML namespaces defined (PR #200, thanks @mephinet for #199).
v2.18.0 (2024-10-15)¶
Python 3.13 (released 2024-10-07) is fully supported (PR #195).
Python 3.8 support is dropped, as it has reached end-of-life (PR #195).
sdmxrequires Python 3.9 or later.
v2.17.0 (2024-09-03)¶
MetadataStructureDefinitionandMetadataSetcan be written to and read from SDMX-ML (PR #192).Clarify differences between
v21.MetadataSet.structured_byandv30.MetadataSet.structured_by, according to the respective standards documents.Read and write
MetadataAttribute,MetadataReport,ReportedAttribute,Period, and associated classes and subclasses.XHTMLAttributeValuecontents are stored aslxml.etreenodes.MetadataStructureDefinition is included when writing
StructureMessage.
Update the base url of the WB_WDI source to use HTTPS instead of plain HTTP (#191, PR #192).
Improvements to
reader.xmlandreader.xml.v21(PR #192).Correctly associate
IteminItemSchemewith its parent, even if the parent is defined after the child (“forward reference”).Bug fix: correctly handle a
MaintainableArtefactthat is explicitly included in a message (that is, not merely referenced), but withis_external_reference = True; the value given in the file is preserved.Bug fix:
FacetValueTypeis written in UpperCamelCase per the standard. The standard specifies lowerCamelCase only in the Information Model.Bug fix: erroneous extra
<Ref style="Ref"/>attribute is no longer written.
Expand logged information in
ComponentList.compare()(PR #192).
v2.16.0 (2024-08-16)¶
New module
sdmx.model.version, classVersion, and convenience functionsversion.increment()andversion.parse()(PR #189).New functions
urn.expand(),urn.normalize(),urn.shorten()and supporting classURN(PR #189).
v2.15.0 (2024-04-28)¶
Adjust the Ten-line usage example for current data returned by ESTAT (#169, PR #170).
Update the base URL of the ILO source (PR #175; thanks @SebaJeku for #177).
StructureMessage.get()can match onIdentifiableArtefact.urn(PR #170). This makes the method more useful in the case that a message includes artefacts with the same ID but differentmaintainerand/orversion.urn.make()can handleDataConsumerScheme,OrganisationScheme,ReportingTaxonomy,TransformationScheme, andVTLMappingScheme(PR #175).New method
StructureMessage.iter_objects()(PR #175).New method
DataMessage.update()(PR #175).Bug fix:
ItemSchemecould not becopy.deepcopy()‘d (PR #170).Bug fix:
TypeErrorwas raised onClient.get()from an SDMX-JSON source (PR #175).
v2.14.0 (2024-02-20)¶
Add support for SDMX 3.0 REST APIs (PR #158). (Note that SDMX-REST v2.1.0 corresponds to version 3.0 of the overall SDMX standards; see the docs.)
Add
v21.URLandv30.URLto construct URLs for different API versions.Add
Resource.availableconstraintand construct (meta)data availability queries per the SDMX 2.1 (REST API v1.5.0) standard (PR #161; thanks @FedorYatsenko).Add
source.Source.versionsattribute to identify REST API versions supported by each source, andSource.get_url_class().Add
ESTAT3, a separate ESTAT data source that makes SDMX 3.0 (REST API v2.1.0) queries.
Add four new ESTAT-related data sources: ESTAT_COMEXT: Eurostat Comext and Prodcom databases and COMP, EMPL, GROW: Directorates General of the European Commission (PR #130).
Automatically handle unsupported values of the
?references=...query parameter for the COMP, EMPL, GROW: Directorates General of the European Commission data sources (#162, PR #163).Bug fix for reading SDMX-ML 2.1: some associations (particularly,
core_representation) not stored correctly if a message contained twoMaintainableArtefactwith the same ID but different maintainer/version (PR #165, thanks @sychsergiy for #164).Drop quirks handling for ILO: International Labour Organization added in v2.6.3 (2022-09-29) (PR #158). The source no longer exhibits the same limitations.
v2.13.1 (2024-01-24)¶
Bug fix for reading
Agencyfrom SDMX-ML 2.1: name of the parentOrganisationwould be incorrectly attached to the Contact (PR #159).Bug fix for writing
Contactto SDMX-ML 2.1:Contact.uriandContact.emailwould be written as, for instance,<str:URI text="https://example.com"/>instead of<str:URI>https://example.com</str:URI>(PR #159).
v2.13.0 (2024-01-23)¶
Expand
modelandreader.xmlsupport for metadata structures and metadata sets (§7 of the Information Model in both SDMX 2.1 and 3.0) (#73, PR #152). This includes the additional classes:model.common:CodingFormatExtendedFacetValueTypeHierarchicalCodeLevel.model.v21:CodelistMapCodeMapDataSetTargetDimensionDescriptorValuesTargetEnumeratedAttributeValueIdentifiableObjectTargetItemAssociationItemSchemeMapMetadataReportMetadataSetMetadataTargetNonEnumeratedAttributeValueOtherNonEnumeratedAttributeValueReportedAttributeReportingCategoryReportingTaxonomyReportPeriodTargetReportStructureStructureSetTargetIdentifiableObjectTargetObjectTargetObjectKeyTargetObjectValueTargetReportPeriodTextAttributeValueXHTMLAttributeValue.model.v30:CodedMetadataAttributeValueIdentifiableObjectSelectionMetadataAttributeDescriptorMetadataAttributeValueMetadataflowMetadataSetMetadataStructureDefinitionOtherUncodedAttributeValueTargetIdentifiableObjectTextAttributeValueUncodedMetadataAttributeValueXHTMLAttributeValue.
New collections on StructureMessage:
hierarchical_codelist,hierarchy,metadatastructure.New class
MetadataMessage.Improve
Structure:New attribute
groupingper the information model.New convenience method
replace_grouping().
reader.xmlparses messages available from ‘actualconstraint’, ‘allowedconstraint’, ‘contentconstraint’, ‘hierarchicalcodelist’, ‘metadatstructure’, ‘structure’, and ‘structureset’ SDMX 2.1 REST API endpoints for all known data sources that support these.Expand explicit marking of particular data sources that do not support the above endpoints.
Add support for validating SDMX-ML messages; see
validate_xml()(#51; thanks @goatsweater for PR #154).sdmxis fully compatible with pandas 2.2.0, released 2024-01-19 (PR #156).
v2.12.1 (2023-12-20)¶
Python 3.12 (released 2023-10-02) is fully supported (PR #145).
Bug fix:
dsd=...argument supplied to the SDMX-ML reader ignored in v2.11.0 and later, causing a warning (PR #147; thanks @miccoli for #146).Bug fix: attribute
<str:Attribute assignmentStatus="…">not read from SDMX-ML (PR #150, thanks @goatsweater for #149).Bug fix: items not written by
writer.xml(PR #150, thanks @goatsweater for #149).Annotation.title→<com:AnnotationTitle>…<com:AnnotationTitle/>.DimensionComponent.order→<str:Dimension position="…">.PrimaryMeasureRelationship→ specific reference to theidof thePrimaryMeasurewithin the associatedDataStructureDefinition.
v2.12.0 (2023-10-11)¶
Fix two bugs in parsing
ItemSchemefrom SDMX-ML:VersionableArtefact.valid_from,valid_tonot stored, andAnnotationsfor the scheme itself erroneously collected by the firstItemin the scheme (PR #143; thanks @goatsweater for #142).Update OECD to support the provider’s recently-added SDMX-ML API (PR #140). Rename the corresponding, older SDMX-JSON source OECD_JSON; work around a known issue with its SSL configuration (see
oecd_json.Client()).
v2.11.0 (2023-08-04)¶
Migration notes¶
As advertised in Migration notes, user code should import either
sdmx.model.v21orsdmx.model.v30. When working with data or structures queried from an SDMX 2.1 or 3.0 data source, be sure to use the corresponding information model (IM). Mixing classes from the two IMs is not supported and may lead to unexpected behaviour.There are several differences between the SDMX 2.1 and 3.0 IMs: the new standards delete some classes, change the name or behaviour of others, and add entirely new classes. (The “Standards” page of the SDMX website includes a link to a document with a “Summary of Changes and New Functionalities”.) User code that functions against
model.v21must be updated if it uses deleted or renamed classes; it may need updating if it depends on behaviour that changes in SDMX 3.0.
All changes¶
Implement the SDMX 3.0 Information Model (
model.v30) and a SDMX-ML 3.0 reader (reader.xml.v30) (PR #135).Add
util.item_structure.parse_item_description(),parse_item(), andparse_all()for handling common, but non-standard “structure expressions” like “A = B + C - D” in thedescriptionofItemsinItemSchemes(or subclasses, such asCodeinCodelist) (#133, PR #137). See examples and further description in the function documentation.Update the ECB: European Central Bank data source URL per a recent change in the service (PR #134).
v2.10.0 (2023-05-20)¶
Switch from third-party
pydanticto Python standard librarydataclasses(PR #128).This is a major change to the
sdmxinternals, but should come with few API changes and some performance improvements. Specific known changes:Individual classes do not have pydantic-supplied
copy()methods. Usecopy.copy()orcopy.deepcopy()from the standard library, as appropriate.Observation.attached_attributevalues should be set explicitly toAttributeValueinstances, rather than to arbitrary types. Instead of:from sdmx.model.v21 import Observation o = obs() o.attached_attribute["CURRENCY"] = "EUR"
…do:
from sdmx.model.v21 import AttributeValue as available o.attached_attribute["CURRENCY"] = av("EUR")
reader.jsonproperly parsesHeader.preparedas adatetimeobject from SDMX-JSON data messages (PR #128).writer.xmlno longer writes objects in a SDMX-MLStructureMessageifMaintainableArtefact.is_external_referenceisTrue(PR #128).Add four new ESTAT-related data sources: ESTAT_COMEXT: Eurostat Comext and Prodcom databases and COMP, EMPL, GROW: Directorates General of the European Commission (PR #130).
Update broken links and other information for some Data sources (PR #130).
Update ABS: Australian Bureau of Statistics (SDMX-ML) to support the ABS’ recently-added “beta” SDMX-ML API (PR #129).
Rename the corresponding SDMX-JSON source ABS_JSON: Australian Bureau of Statistics (SDMX-JSON), update web service URL and quirks handling (
abs_json.Source) (PR #129, PR #130).
v2.9.0 (2023-04-30)¶
Add
sdmx.to_csv()(writer.csv) to generate SDMX-CSV 1.0 (corresponding to SDMX 2.1) representation ofDataSets(#36, PR #125).Information Model classes (PR #125):
Add
AnnotableArtefact.eval_annotation(), which can be used to retrieve Python data structures stored usingrepr()asAnnotation.texton an object.Implement
KeyValue.__lt__(), for use with Pythonsorted().Implement
DataSet.__str__(). The previous default string representation included the representation of every observation in the data set, which could be excessively verbose. Userepr(ds)explicitly if this is desired.ComponentList.append()(thus also child classes includingDimensionDescriptor) now setsDimensionComponent.orderon the appended components (dimensions), if not already set.
-
Write
DataSet.attrib—that is,AttributeValueattached directly to a data set—rather than to its contents.Write
Contact, for instance within anAgencyScheme.
Bug fix: correctly handle
&detail=referencepartialREST query parameter andStructureMessagecontaining ≥2MaintainableArtefactwith the same maintainer and ID, but different versions (#116, PR #124). See the documentation forreader.xml.sdmxis fully compatible with pandas 2.0.0, released 2023-04-03 (PR #124). The minimum version of Python is increased from 3.7 (EOL 2023-06-27) to 3.8.
v2.8.0 (2023-03-31)¶
Migration notes¶
In order to prepare for future support of SDMX 3.0, code such as the following will emit a DeprecationWarning:
from sdmx.model import DataStructureDefinition
from sdmx import model
dsd = model.DataStructureDefinition(...)
This occurs for sdmx.model classes (for instance v21.DataStructureDefinition) which may have a different implementation in SDMX 3.0 than in SDMX 2.1.
It does not occur for classes that are unchanged from SDMX 2.1 to 3.0, for instance InternationalString.
Code can be adjusted by importing explicitly from the new model.v21 submodule:
from sdmx.model.v21 import DataStructureDefinition
from sdmx.model import v21 as model
dsd = model.DataStructureDefinition(...)
All changes¶
Outline and prepare for for SDMX 3.0 support (PR #120). Read Standards versions for details.
The internal
Formatis replaced by aMediaType, allowing to distinguish the “, version=3.0.0” parameters in the HTTPContent-Typeheader.xml.v21.Reader.media_typesandjson.Reader.media_typesexplicitly indicate supported media types.ItemScheme.is_partialdefaults toNone.Add empty/stub
format.csv,reader.csv(cf. #34), andmodel.v30.Improve readability in Implementation notes (PR #121).
v2.7.1 (2023-03-09)¶
v2.7.0 (2022-11-14)¶
Python 3.11 is fully supported (PR #109).
Changes for specific data sources:
ESTAT: Eurostat and related: update web service URL, quirks handling (
estat.Source), tests, and usage throughout documentation (PR #107, PR #109, thanks @zymon).International Monetary Fund: work around #102 (thanks @zymon), an error in some structure messages (PR #103).
ISTAT: National Institute of Statistics (Italy): update web service URL (PR #105; thanks @miccoli for #104).
Add
MetadataflowDefinition,MetadataStructureDefinition, and handle references to these inreader.xml(PR #105).Handle SDMX-ML observed in the wild (PR #109):
Elements that normally contain text but appear without even a text node, such as
<com:AnnotationURL/>.XML namespaces defined on the message element such as
<mes:StructureSpecificData xmlns:u="...">followed by<u:DataSet>instead of<mes:DataSet>.
Use the user-supplied
dsd=...argument toClient.get(), even if its ID does not match those used locally in an SDMX-MLDataMessage(PR #106, #104).Expand the source/endpoint test matrix (PR #109). Every REST API endpoint is queried for every data source, even if it is known to be not implemented. This allows to spot when source implementations change.
Sort entries in
sources.json(PR #109).
v2.6.3 (2022-09-29)¶
Update ILO: International Labour Organization web service URL and quirks handling (PR #97, thanks @ethangelbach).
Use HTTPS for ESTAT: Eurostat and related (PR #97).
Bump minimum version of
pydanticto 1.9.2 (PR #98).Always return all objects parsed from a SDMX-ML
StructureMessage(PR #99).If two or more
MaintainableArtefacthave the same ID (for example, “CL_FOO”);sdmxwould formerly store only the last one parsed. Now, each is returned, with keys like{maintainer's id}:{object id}such as would appear in an SDMX URI; for example, “AGENCY_A:CL_FOO”, “AGENCY_B:CL_FOO”, etc.Recognize the MIME type
application/vnd.sdmx.generic+xml;version=2.1(PR #99).Catch some cases where
nameanddescriptionwere discarded when parsing SDMX-ML (PR #99).
v2.6.2 (2022-01-11)¶
This release contains mainly compatibility updates and testing changes.
https://khaeru.github.io/sdmx/ now serves a dashboard summarizing automatic, daily tests of every SDMX 2.1 REST API endpoints for every data source built-in to
sdmx. See Handling and testing limitations and (un)supported endpoints (PR #90).Pydantic >= 1.9 is supported (PR #91).
Python 3.10 is fully supported (PR #89).
v2.6.1 (2021-07-27)¶
Bug fixes¶
reader.xmlignored values like0or0.0that evaluated equivalent toFalse(PR #86).
v2.6.0 (2021-07-11)¶
Expand documentation of Handling and testing limitations and (un)supported endpoints; add a large number of expected test failures for limitations of specific web services (PR #84).
Add information from the SDMX-REST standard (PR #84):
format.FORMATS, all media (MIME or content) types and their attributes.Resource, expanded and including all resource names appearing in the standard.rest.RESPONSE_CODE.
Information Model pieces (PR #84):
Classes
DataConsumerandDataProvider, including reading these from SDMX-ML.Attribute
DataSet.described_by, referencing aDFDthat in the same waystructured_byreferences aDSD.
-
Write
Footerinto messages.Do not create URNs for members of
ItemSchemes; only write existing URNs. This improves round-trip fidelity to original files.
Convenience methods and functionality (PR #84):
StructureMessage.objects()to access collections of structures using a class reference.len()onMemberSelection.model.get_class()now works withResourceenumeration values as arguments.
Internal (PR #84):
New
BaseReadermethodssupports_content_type()andsupports_suffix().Improve typing.
Expand test coverage.
v2.5.0 (2021-06-27)¶
Add BBK: German Federal Bank and BIS: Bank for International Settlements services to supported sources (PR #83).
Work around some non-standard behaviours of
BBK; see #82.
Document how Countdown to 2030 data can be accessed from the UNICEF service (PR #83).
Reduce noise when
requests_cacheis not installed (#75, PR #80). An exception is still raised if (a) the package is not installed and (b) cache-related arguments are passed toClient.Bug fix:
verify=Falsewas not passed to the preliminary request used to validate adictkey for a data request (PR #80; thanks @albertame for #77).Handle
<mes:Department>and<mes:Role>in SDMX-ML headers (#78, PR #79).
v2.4.1 (2021-04-12)¶
v2.4.0 (2021-03-28)¶
IdentifiableArtefactcan besorted()(PR #71).Add
DataStructureDefinition.iter_keys()to iterate over valid keys, optionally with av21.Constraint(PR #72)Also add
ContentConstraint.iter_keys(),DataflowDefinition.iter_keys().Implement or improve
Constraint.__contains__(),CubeRegion.__contains__(),ContentConstraint.__contains__(),v21.KeyValue.__eq__(), andKey.__eq__().
Speed up creation of
Keyobjects by improvingpydanticusage, updatingKey.__init__(), and addingKey._fast().Simplify
.validate_dictlike; adddictlike_field, and simplifypydanticvalidation ofDictLikeobjects, keys, and values.
v2.3.0 (2021-03-10)¶
Improve typing of
Itemand subclasses such asCode(PR #66).parentandchildelements are typed the same as a subclass.Require
pydantic>= 1.8.1, and remove workarounds for limitations in earlier versions (PR #66).The default branch of the
sdmxGitHub repository is renamedmain.
Bug fixes¶
v2.2.1 (2021-02-27)¶
Temporary exclude
pydanticversions >= 1.8 (PR #62).
v2.2.0 (2021-02-26)¶
New convenience method
AnnotableArtefact.get_annotation()to return but not remove an Annotation, for instance by its ID (PR #60).Add
py.typedto support type checking (with mypy) in packages that depend onsdmx.
v2.1.0 (2021-02-22)¶
ItemScheme.append()now raisesValueErroron duplicate IDs (PR #58).Item.parentstores a reference to the containingItemSchemefor top-level Items that have no hierarchy/parent of their own. This allows navigating from any Item to the ItemScheme that contains it.Item.get_scheme()is added as a convenience method (PR #58).reader.xmlinternals reworked for significant speedups in parsing of SDMX-ML (PR #58).New convenience method
StructureMessage.get()to retrieve objects by ID across the multiple collections in StructureMessage (PR #58).New convenience method
AnnotableArtefact.pop_annotation()to locate, remove, and return a Annotation, for instance by its ID (PR #58).len()of aDataKeySetgives the length ofDataKeySet.keys(PR #58).
v2.0.1 (2021-01-31)¶
Bug fixes¶
NoSpecifiedRelationshipandPrimaryMeasureRelationshipdo not need to be instantiated; they are singletons (#54, PR #56).attributes= “d” ignored in
to_pandas()(#55, PR #56).
v2.0.0 (2021-01-26)¶
Migration notes¶
Code that calls Request() emits DeprecationWarning
and logs a message with level WARNING:
>>> sdmx.Request("ECB")
Request class will be removed in v3.0; use Client(...)
<sdmx.client.Client object at 0x7f98787e7d60>
Instead, use:
sdmx.Client("ECB")
Per the standard semantic versioning approach, this feature is marked as deprecated in version 2.0, and will be removed no sooner than version 3.0.
References to sdmx.logger should be updated to sdmx.log.
Instead of passing the log_level parameter to Client,
access this standard Python Logger and change its level,
as described at HOWTO control logging.
All changes¶
The large library of test specimens for
sdmxis no longer shipped with the package, reducing the archive size by about 80% (#18, PR #52). The specimens can be retrieved for running tests locally; see Testing.The
Requestclass is renamedClientfor semantic clarity (#11, PR #44):A Client can open a
requests.Sessionand might make manyrequests.Requestsagainst the same web service.The log_level parameter to
Clientis deprecated.Some internal modules are renamed. These should not affect user code; if they do, adjust that code to use the top-level objects.
sdmx.apiis renamedsdmx.client.sdmx.remoteis renamedsdmx.session.sdmx.reader.sdmxmlis renamedsdmx.reader.xml, to conform withsdmx.format.xmlandsdmx.writer.xml.sdmx.reader.sdmxjsonis renamedsdmx.reader.json.
v1.7 and earlier¶
v1.7.0 (2021-01-26)¶
New features¶
Add The Pacific Community’s Pacific Data Hub as a data source (PR #30).
Add classes to
sdmx.model:v21.TimeRangeValue,Period,RangePeriod, and parse<com:TimeRange>and related tags in SDMX-ML (PR #30).
Bug fixes¶
Output SDMX-ML header elements in order expected by standard XSD (#42, PR #43).
Respect override argument to
add_source()(PR #41).
v1.6.0 (2020-12-16)¶
New features¶
Support Python 3.9 (using pydantic ≥ 1.7) (PR #37).
Add National Bank of Belgium as a data source (PR #32).
Add Statistics Lithuania as a data source (PR #33).
Bug fixes¶
Data set-level attributes were not collected by
sdmxml.Reader(#29, PR #33).
v1.5.0 (2020-11-12)¶
Add a brief tutorial on creating SDMX-ML messages from pure Python objects (#23, PR #24).
Add Statistics Estonia as a data source (PR #25).
Supply provider=“ALL” to INSEE structure queries by default (#21, PR #22)
v1.4.0 (2020-08-17)¶
New features¶
Enhance
to_xml()to handleDataMessages(PR #13).In v1.4.0, this feature supports a subset of DataMessages and DataSets. If you have an example of a DataMessages that
sdmx1.4.0 cannot write, please file an issue on GitHub with a file attachment. SDMX-ML features used in such examples will be prioritized for future improvements.Add
compare()methods toDataMessage,DataSet, and related classes (PR #13).
Bug fixes¶
Fix parsing of
MeasureDimensionreturned by SGR for data structure queries (PR #14).
v1.3.0 (2020-08-02)¶
Adjust imports for compatibility with pandas 1.1.0 (PR #10).
Add World Bank World Development Indicators (WDI) service to supported sources (PR #10).
v1.2.0 (2020-06-04)¶
New features¶
Methods like
IdentifiableArtefact.compareare added for recursive comparison ofmodelobjects (PR #6).to_xml()covers a larger subset of SDMX-ML, including almost all contents of aStructureMessage(PR #6).
v1.1.0 (2020-05-18)¶
Data model changes¶
…to bring sdmx into closer alignment with the standard Information Model (PR #4):
Change
Header.receiverandHeader.senderto optionalAgency, notstr.Add
Header.sourceandtest.IdentifiableArtefact.idis strictly typed asstr, with a singleton object (analogous toNone) used for missing IDs.IdentifiableArtefact.id,VersionableArtefact.version, andMaintainableArtefact.maintainerare inferred from a URN if one is passed during construction.VersionableArtefact.identicalandMaintainableArtefact.identicalcompare on version and maintainer attributes, respectively.Facet,Representation, andISOConceptReferenceare strictly validated and cannot be assigned non-IM attributes.Add
OrganisationScheme,NoSpecifiedRelationship,PrimaryMeasureRelationship,DimensionRelationship, andGroupRelationshipas distinct classes.Type of
DimensionRelationship.dimensionsisDimensionComponent, not the narrowerDimension.v21.DataStructureDefinition.measuresis an emptyv21.MeasureDescriptorby default, notNone.DataSet.add_obsnow acceptsObservationswith noSeriesKeyassociation, and sets this association to the one provided as an argument.String representations are simplified but contain more information.
New features¶
Item.hierarchical_idandItemScheme.get_hierarchical()create and search on IDs like ‘A.B.C’ for Item ‘A’ with child/grandchild Items ‘B’ and ‘C’ (PR #4).New methods
.parent_class,get_reader_for_path(),detect_content_reader(), and.reader.register(PR #4)..sdmxml.Readeruses an event-driven, rather than recursive/tree iterating, parser (PR #4).The codebase is improved to pass static type checking with mypy (PR #4).
Add
to_xml()to generate SDMX-ML for a subset of the IM (PR #3).
Test suite¶
PR #2: Add tests of data queries for source(s): OECD
v1.0.0 (2020-05-01)¶
Project forked and renamed to
sdmx(module) /sdmx1(on PyPI, due to an older, unmaintained package with the same name).sdmx.modelis reimplemented.Python typing and pydantic are used to force tight compliance with the SDMX Information Model (IM). Users familiar with the IM can use
sdmxwithout the need to understand implementation-specific details.IM classes are no longer tied to
sdmx.readerinstances and can be created and manipulated outside of a read operation.
sdmx.apiandsdmx.remoteare reimplemented to (1) match the semantics of the requests package and (2) be much thinner.Data sources are modularized in
Source.Idiosyncrasies of particular data sources (such as ESTAT’s process for large requests) are handled by source-specific subclasses. As a result,
sdmx.apiis leaner.
Testing coverage is significantly expanded.
Promised, but untested, features of the 0.x series now have tests, to ensure feature parity.
There are tests for each data source (
tests/test_sources.py`) to ensure the package can handle idiosyncratic behaviour.The pytest-remotedata pytest plugin allows developers and users to run or skip network tests with –remote-data.
Breaking changes¶
Python 3.6 and earlier (including Python 2) are not supported.
Migrating¶
Writer.write(..., reverse_obs=True): use the standard pandas indexing approach to reverse a pd.Series:s.iloc[::-1].odo support is no longer built-in; however, users can still register a SDMX resource with odo. See the HOWTO.
write_dataset: the parse_time and fromfreq arguments are replaced by datetime; see the method documentation and the walkthrough section for examples.
pandaSDMX (versions 0.9 and earlier)¶
pandaSDMX v0.9 (2018-04)¶
This version is the last tested on Python 2.x. Future versions will be tested on Python 3.5+ only
New features¶
four new data providers INEGI (Mexico), Norges Bank (Norway), International Labour Organization (ILO) and Italian statistics office (ISTAT)
model: make Ref instances callable for resolving them, i.e. getting the referenced object by making a remote request if needed
improve loading of structure-specific messages when DSD is not passed / must be requested on the fly
process multiple and cascading content constraints as described in the Technical Guide (Chap. 6 of the SDMX 2.1 standard)
StructureMessages and DataMessages now have properties to compute the constrained and unconstrained codelists as dicts of frozensets of codes. For DataMessage this is useful when
series_keyswas set to True when making the request. This prompts the data provider to generate a dataset without data, but with the complete set of series keys. This is the most accurate representation of the available series. Agencies such as IMF and ECB support this feature.
v0.8.2 (2017-12-21)¶
fix reading of structure-specific data sets when DSD_ID is present in the data set
v0.8.1 (2017-12-20)¶
fix broken package preventing pip installs of the wheel
v0.8 (2017-12-12)¶
add support for an alternative data set format defined for SDMXML messages. These so-called structure-specific data sets lend themselves for large data queries. File sizes are typically about 60 % smaller than with equivalent generic data sets. To make use of structure-specific data sets, instantiate Request objects with agency IDs such as ‘ECB_S’, ‘INSEE_S’ or ‘ESTAT_S’ instead of ‘ECB’ etc. These alternative agency profiles prompt pandaSDMX to execute data queries for structure-specific data sets. For all other queries they behave exactly as their siblings. See a code example in chapter 5 of the docs.
raise ValueError when user attempts to request a resource other than data from an agency delivering data in SCMX-JSON format only (OECD and ABS).
Update INSEE profile
handle empty series properly
data2pd writer: the code for Series index generation was rewritten from scratch to make better use of pandas’ time series functionality. However, some data sets, in particular from INSEE, which come with bimonthly or semestrial frequencies cannot be rendered as PeriodIndex. Pass
parse_time=Falseto the .write method to prevent errors.
v0.7.0 (2017-06-10)¶
add new data providers:
Australian Bureau of Statistics
International Monetary Fund - SDMXCentral only
United Nations Division of Statistics
UNESCO (free registration required)
World Bank - World Integrated Trade Solution (WITS)
new feature: load metadata on data providers from json file; allow the user to add new agencies on the fly by specifying an appropriate JSON file using the
pandasdmx.api.Request.load_agency_profile.new
pandasdmx.api.Request.preview_dataproviding a powerful fine-grain key validation algorithm by downloading all series-keys of a dataset and exposing them as a pandas DataFrame which is then mapped to the cartesian product of the given dimension values. Works only with data providers such as ECB and UNSD which support “series-keys-only” requests. This feature could be wrapped by a browser-based UI for building queries.SDMX-JSON reader: add support for flat and cross-sectional datasets, preserve dimension order where possible
structure2pd writer: in codelists, output Concept rather than Code attributes in the first line of each code-list. This may provide more information.
v0.6.1 (2017-02-03)¶
fix 2to3 issue which caused crashes on Python 2.7
v0.6 (2017-01-07)¶
This release contains some important stability improvements.
Bug fixes¶
JSON data from OECD is now properly downloaded
The data writer tries to glean a frequency value for a time series from its attributes. This is helpful when exporting data sets, e.g., from INSEE (Issue 41).
Known issues¶
A data set which lacks a FREQ dimension or attribute can be exported as pandas DataFrame only when parse_time=False?, i.e. no DateTime index is generated. The resulting DataFrame has a string index. Use pandas magic to create a DateTimeIndex from there.
v0.5 (2016-10-30)¶
New features¶
new reader module for SDMX JSON data messages
add OECD as data provider (data messages only)
pandasdmx.model.Categoryis now an iterator over categorised objects. This greatly simplifies category usage. Besides, categories with the same ID while belonging to multiple category schemes are no longer conflated.
API changes¶
Request constructor: make agency ID case-insensitive
As
Categoryis now an iterator over categorised objects,Categorisationsis no longer considered part of the public API.
Bug fixes¶
SDMX-ML reader: fix AttributeError in write_source method, thanks to Topas
correctly distinguish between categories with same ID while belonging to different category schemes
v0.4 (2016-04-11)¶
New features¶
add new provider INSEE, the French statistics office (thanks to Stéphan Rault)
register ‘.sdmx’ files with Odo if available
logging of http requests and file operations.
new structure2pd writer to export codelists, dataflow-definitions and other structural metadata from structure messages as multi-indexed pandas DataFrames. Desired attributes can be specified and are represented by columns.
API changes¶
pandasdmx.api.Requestconstructor accepts alog_levelkeyword argument which can be set to a log-level for the pandasdmx logger and its children (currently only pandasdmx.api)pandasdmx.api.Requestnow has atimeoutproperty to set the timeout for http requestsextend api.Request._agencies configuration to specify agency- and resource-specific settings such as headers. Future versions may exploit this to provide reader selection information.
api.Request.get: specify http_headers per request. Defaults are set according to agency configuration
Response instances expose Message attributes to make application code more succinct
rename
pandasdmx.api.Messageattributes to singular form. Old names are deprecated and will be removed in the future.pandasdmx.api.Requestexposes resource names such as data, datastructure, dataflow etc. as descriptors calling ‘get’ without specifying the resource type as string. In interactive environments, this saves typing and enables code completion.data2pd writer: return attributes as namedtuples rather than dict
use patched version of namedtuple that accepts non-identifier strings as field names and makes all fields accessible through dict syntax.
remove GenericDataSet and GenericDataMessage. Use DataSet and DataMessage instead
sdmxml reader: return strings or unicode strings instead of LXML smart strings
sdmxml reader: remove most of the specialized read methods. Adapt model to use generalized methods. This makes code more maintainable.
sdmx.model.Representationfor DSD attributes and dimensions now supports text not just code lists.
Other changes and enhancements¶
documentation has been overhauled. Code examples are now much simpler thanks to the new structure2pd writer
testing: switch from nose to py.test
improve packaging. Include tests in sdist only
numerous bug fixes
v0.3.1 (2015-10-04)¶
This release fixes a few bugs which caused crashes in some situations.
v0.3.0 (2015-09-22)¶
support for requests-cache allowing to cache SDMX messages in memory, MongoDB, Redis or SQLite.
pythonic selection of series when requesting a dataset: Request.get allows the
keykeyword argument in a data request to be a dict mapping dimension names to values. In this case, the dataflow definition and datastructure definition, and content-constraint are downloaded on the fly, cached in memory and used to validate the keys. The dotted key string needed to construct the URL will be generated automatically.The Response.write method takes a
parse_timekeyword arg. Set it to False to avoid parsing of dates, times and time periods as exotic formats may cause crashes.The Request.get method takes a
memcachekeyword argument. If set to a string, the received Response instance will be stored in the dictRequest.cachefor later use. This is useful when, e.g., a DSD is needed multiple times to validate keys.fixed base URL for Eurostat
major refactorings to enhance code maintainability
v0.2.2¶
Make HTTP connections configurable by exposing the requests.get API through the
pandasdmx.api.Requestconstructor. Hence, proxy servers, authorisation information and other HTTP-related parameters consumed byrequests.getcan be specified for eachRequestinstance and used in subsequent requests. The configuration is exposed as a dict through a newRequest.client.configattribute.Responses have a new
http_headersattribute containing the HTTP headers returned by the SDMX server
v0.2.1¶
Request.get: allow fromfile to be a file-like object
extract SDMX messages from zip archives if given. Important for large datasets from Eurostat
automatically get a resource at an URL given in the footer of the received message. This allows to automatically get large datasets from Eurostat that have been made available at the given URL. The number of attempts and the time to wait before each request are configurable via the
get_footer_urlargument.
v0.2.0 (2015-04-13)¶
This version is a quantum leap. The whole project has been redesigned and rewritten from scratch to provide robust support for many SDMX features. The new architecture is centered around a pythonic representation of the SDMX information model. It is extensible through readers and writers for alternative input and output formats. Export to pandas has been dramatically improved. Sphinx documentation has been added.
v0.1.2 (2014-09-17)¶
fix xml encoding. This brings dramatic speedups when downloading and parsing data
extend description.rst
v0.1 (2014-09)¶
Initial release