test_filter_ava_registration_authority_1 fails on i586

[mcepl]

When packaging pysaml2 for OpenSUSE on i586 (tests on other 32bit architecture is still pending) it fails in test_filter_ava_registration_authority_1 test:

[  185s] ___________________ test_filter_ava_registration_authority_1 ___________________
[  185s] 
[  185s]     def test_filter_ava_registration_authority_1():
[  185s]         mds = MetadataStore(ATTRCONV, sec_config, disable_ssl_certificate_validation=True)
[  185s]         mds.imp(METADATACONF["1"])
[  185s]     
[  185s]         policy_conf = {
[  185s]             "default": {
[  185s]                 "lifetime": {"minutes": 15},
[  185s]                 "attribute_restrictions": None,
[  185s]             },
[  185s]             "http://rr.aai.switch.ch/": {
[  185s]                 "attribute_restrictions": {
[  185s]                     "givenName": None,
[  185s]                     "surName": None,
[  185s]                 }
[  185s]             }
[  185s]         }
[  185s]         policy = Policy(restrictions=policy_conf, mds=mds)
[  185s]     
[  185s]         attributes = {
[  185s]             "givenName": ["Derek"],
[  185s]             "surName": ["Jeter"],
[  185s]             "mail": [
[  185s]                 "derek@nyy.mlb.com",
[  185s]                 "dj@example.com",
[  185s]             ],
[  185s]         }
[  185s]     
[  185s]         # SP registered with http://rr.aai.switch.ch/
[  185s]         ava = policy.filter(attributes, "https://aai-idp.unibe.ch/idp/shibboleth")
[  185s] >       assert _eq(sorted(list(ava.keys())), ["givenName", "surName"])
[  185s] E       AssertionError: assert False
[  185s] E        +  where False = _eq(['givenName', 'mail', 'surName'], ['givenName', 'surName'])
[  185s] E        +    where ['givenName', 'mail', 'surName'] = sorted(['givenName', 'surName', 'mail'])
[  185s] E        +      where ['givenName', 'surName', 'mail'] = list(dict_keys(['givenName', 'surName', 'mail']))
[  185s] E        +        where dict_keys(['givenName', 'surName', 'mail']) = <built-in method keys of dict object at 0xf40bfed8>()
[  185s] E        +          where <built-in method keys of dict object at 0xf40bfed8> = {'givenName': ['Derek'], 'mail': ['derek@nyy.mlb.com', 'dj@example.com'], 'surName': ['Jeter']}.keys
[  185s] 
[  185s] tests/test_20_assertion.py:928: AssertionError

Code Version

v6.3.1 from tarball (no patches applied)
Python 3.8
All other details are in complete build log.

[c00kiemon5ter]

If you look at the build log you'll see

ERROR    saml2.mdstore:mdstore.py:625 Invalid XML message: Class 'EntitiesDescriptor' instance: Wrong type of value '2999-12-13T10:05:05Z' on attribute 'valid_until' expected it to be dateTime

2999-12-13T10:05:05Z is actually a valid xs:dateTime value, but the value cannot be represented in 32bits as a number. While the Python datetime module and class can represent this correctly on such platforms, the way the code parses and converts the value is problematic and fails. There is a (big) PR that fixes this but has not been merged yet.

As a workaround, changing the xml to have a different value (anything under 2038-01-19T03:14:07Z) should make this error go away. However, this is not a single case - more such errors will occur.

FAILED tests/test_20_assertion.py::test_filter_ava_registration_authority_1 - AssertionError: assert False
FAILED tests/test_30_mdstore.py::test_incommon_1 - assert 0 > 1700
FAILED tests/test_30_mdstore.py::test_switch_1 - assert 0 > 160
FAILED tests/test_30_mdstore.py::test_load_extern_incommon - assert 0
FAILED tests/test_30_mdstore.py::test_registration_info - AssertionError: assert 'http://rr.aai.switch.ch/' == None
FAILED tests/test_30_mdstore.py::test_registration_info_no_policy - AssertionError: assert 'http://eduid.hu' == None
FAILED tests/test_30_mdstore_old.py::test_incommon_1 - assert 0 > 1700
FAILED tests/test_30_mdstore_old.py::test_switch_1 - assert 0 > 160
FAILED tests/test_30_mdstore_old.py::test_load_external - assert 0 > 1
FAILED tests/test_31_config.py::test_assertion_consumer_service - saml2.s_utils.UnknownSystemEntity: https://www.zimride.com/shibboleth
FAILED tests/test_38_metadata_filter.py::test_swamid_sp - assert 0 == 417
FAILED tests/test_38_metadata_filter.py::test_swamid_idp - assert 0 == 275
FAILED tests/test_41_response.py::TestResponse::test_other_response - AttributeError: 'NoneType' object has no attribute 'assertion'
FAILED tests/test_62_vo.py::TestVirtualOrg::test_mta - KeyError: 'urn:mace:example.com:it:tek'
FAILED tests/test_62_vo.py::TestVirtualOrg::test_unknown_subject - KeyError: 'urn:mace:example.com:it:tek'
FAILED tests/test_62_vo.py::TestVirtualOrg_2::test_mta - KeyError: 'urn:mace:example.com:it:tek'
FAILED tests/test_62_vo.py::TestVirtualOrg_2::test_unknown_subject - KeyError: 'urn:mace:example.com:it:tek'

The main issue is that those entries are not registered as valid metadata internally and metadata-related checks start failing.
I will try to have a more thorough quick look.

[c00kiemon5ter]

Indeed these are all errors because of the conversion of the date into a number that cannot fit in 32bits. Applying the diff bellow fixes the errors.

diff --git a/tests/InCommon-metadata.xml b/tests/InCommon-metadata.xml
index 405ec8c3..3615981e 100644
--- a/tests/InCommon-metadata.xml
+++ b/tests/InCommon-metadata.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="UTF-8"?><EntitiesDescriptor xmlns="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:shibmd="urn:mace:shibboleth:metadata:1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ID="INC20140204T195141" Name="urn:mace:incommon" validUntil="2999-02-18T10:00:00Z" xsi:schemaLocation="urn:oasis:names:tc:SAML:2.0:metadata sstc-saml-schema-metadata-2.0.xsd urn:mace:shibboleth:metadata:1.0 shibboleth-metadata-1.0.xsd http://www.w3.org/2000/09/xmldsig# xmldsig-core-schema.xsd"><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
+<?xml version="1.0" encoding="UTF-8"?><EntitiesDescriptor xmlns="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:shibmd="urn:mace:shibboleth:metadata:1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ID="INC20140204T195141" Name="urn:mace:incommon" validUntil="2036-02-18T10:00:00Z" xsi:schemaLocation="urn:oasis:names:tc:SAML:2.0:metadata sstc-saml-schema-metadata-2.0.xsd urn:mace:shibboleth:metadata:1.0 shibboleth-metadata-1.0.xsd http://www.w3.org/2000/09/xmldsig# xmldsig-core-schema.xsd"><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
 <ds:SignedInfo>
 <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
 <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
diff --git a/tests/attribute_response.xml b/tests/attribute_response.xml
index 9a7aa03b..33f1118c 100644
--- a/tests/attribute_response.xml
+++ b/tests/attribute_response.xml
@@ -32,13 +32,13 @@
                     Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
                 <saml2:SubjectConfirmationData Address="192.168.1.1"
                                                InResponseTo="id-f4d370f3d03650f3ec0da694e2348bfe"
-                                               NotOnOrAfter="2999-09-14T21:06:32.081Z"
+                                               NotOnOrAfter="2036-09-14T21:06:32.081Z"
                                                Recipient="https://myreviewroom.com/saml2/acs/"
                         />
             </saml2:SubjectConfirmation>
         </saml2:Subject>
         <saml2:Conditions NotBefore="2014-09-14T21:01:32.081Z"
-                          NotOnOrAfter="2999-09-14T21:06:32.081Z"
+                          NotOnOrAfter="2036-09-14T21:06:32.081Z"
                 >
             <saml2:AudienceRestriction>
                 <saml2:Audience>urn:mace:example.com:saml:roland:sp
diff --git a/tests/idp_example.xml b/tests/idp_example.xml
index 6c7efca9..dd7bc86f 100644
--- a/tests/idp_example.xml
+++ b/tests/idp_example.xml
@@ -2,7 +2,7 @@
 <ns0:EntityDescriptor xmlns:ns0="urn:oasis:names:tc:SAML:2.0:metadata"
                       xmlns:ns1="http://www.w3.org/2000/09/xmldsig#"
                       entityID="http://localhost:8088/idp.xml"
-                      validUntil="2014-04-12T06:06:13Z">
+                      validUntil="2036-04-12T06:06:13Z">
     <ns0:IDPSSODescriptor WantAuthnRequestsOnlyWithValidCert="false"
                           WantAuthnRequestsSigned="false"
                           protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
diff --git a/tests/metadata.aaitest.xml b/tests/metadata.aaitest.xml
index 5a1f2bd2..298afc81 100644
--- a/tests/metadata.aaitest.xml
+++ b/tests/metadata.aaitest.xml
@@ -7,7 +7,7 @@
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     ID="AAITest-20140205105921"
                     Name="urn:mace:switch.ch:aaitest"
-                    validUntil="2999-02-10T09:59:21Z"
+                    validUntil="2036-02-10T09:59:21Z"
                     xsi:schemaLocation="urn:oasis:names:tc:SAML:2.0:metadata saml-schema-metadata-2.0.xsd urn:mace:shibboleth:metadata:1.0 shibboleth-metadata-1.0.xsd http://www.w3.org/2000/09/xmldsig# xmldsig-core-schema.xsd">
     <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
         <ds:SignedInfo>
diff --git a/tests/metadata.xml b/tests/metadata.xml
index 2ecd15a2..90fb3d2d 100644
--- a/tests/metadata.xml
+++ b/tests/metadata.xml
@@ -1,6 +1,6 @@
 <?xml version='1.0' encoding='UTF-8'?>
 <ns0:EntitiesDescriptor name="urn:mace:example.com:saml:test"
-                        validUntil="2999-12-04T17:31:07Z"
+                        validUntil="2036-12-04T17:31:07Z"
                         xmlns:ns0="urn:oasis:names:tc:SAML:2.0:metadata">
   <ns0:EntityDescriptor entityID="urn:mace:example.com:saml:roland:sp">
     <ns0:SPSSODescriptor AuthnRequestsSigned="False" WantAssertionsSigned="True"
diff --git a/tests/metasp.xml b/tests/metasp.xml
index b166e21f..b1756d59 100644
--- a/tests/metasp.xml
+++ b/tests/metasp.xml
@@ -1,5 +1,5 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<ns0:EntitiesDescriptor xmlns:ns0="urn:oasis:names:tc:SAML:2.0:metadata" name="urn:mace:umu.se:saml:test" validUntil="2010-12-01T09:22:16Z">
+<ns0:EntitiesDescriptor xmlns:ns0="urn:oasis:names:tc:SAML:2.0:metadata" name="urn:mace:umu.se:saml:test" validUntil="2036-12-01T09:22:16Z">
   <ns0:EntityDescriptor entityID="urn:mace:umu.se:saml:roland:sp">
     <ns0:SPSSODescriptor AuthnRequestsSigned="False" WantAssertionsSigned="True" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
       <ns0:KeyDescriptor>
diff --git a/tests/remote_data/InCommon-metadata-export.xml b/tests/remote_data/InCommon-metadata-export.xml
index 5ac71b11..4c45785c 100644
--- a/tests/remote_data/InCommon-metadata-export.xml
+++ b/tests/remote_data/InCommon-metadata-export.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?><EntitiesDescriptor xmlns="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:shibmd="urn:mace:shibboleth:metadata:1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ID="_20191126T193752" validUntil="3001-01-01T00:00:00Z">
+<?xml version="1.0" encoding="UTF-8" standalone="no"?><EntitiesDescriptor xmlns="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:shibmd="urn:mace:shibboleth:metadata:1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ID="_20191126T193752" validUntil="2036-01-01T00:00:00Z">
 <md:Extensions xmlns:mdrpi="urn:oasis:names:tc:SAML:metadata:rpi">
 <mdrpi:PublicationInfo creationInstant="2019-11-26T19:37:52Z" publisher="https://incommon.org"/>
 </md:Extensions>
diff --git a/tests/swamid-2.0.xml b/tests/swamid-2.0.xml
index 964e9ef5..ee4594ae 100644
--- a/tests/swamid-2.0.xml
+++ b/tests/swamid-2.0.xml
@@ -4,7 +4,7 @@
       This file was automatically generated - do not edit
 
     -->
-<md:EntitiesDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:shibmd="urn:mace:shibboleth:metadata:1.0" Name="http://md.swamid.se/md/swamid-2.0.xml" cacheDuration="PT8H" validUntil="2999-12-13T10:05:05Z"><ds:Signature><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>wdokYz5tEa8aCh+fEPqytS/y9W8=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>Wsr/+VrII3wK5rAzQKmjSzflCIuIhQboSc6sIxQwWw3toALPfY5fBl1XHPKGFXxY
+<md:EntitiesDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:shibmd="urn:mace:shibboleth:metadata:1.0" Name="http://md.swamid.se/md/swamid-2.0.xml" cacheDuration="PT8H" validUntil="2036-12-13T10:05:05Z"><ds:Signature><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>wdokYz5tEa8aCh+fEPqytS/y9W8=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>Wsr/+VrII3wK5rAzQKmjSzflCIuIhQboSc6sIxQwWw3toALPfY5fBl1XHPKGFXxY
 t6W7MxPjU1FKh4PdISeTgqYUwvrX2YDE7HxoYAZR5n5cZlogBZIR3dUwXAcJ75pX
 tDUHpoqNyEJgoaeTiFhNBrfwGPlWNb0RstfM+iMIpdNTlSFHvuHMxkJSEunjzbcj
 7OU8KcYSlosw4wqdI/G50aQAjSJf+M1wARHtbPvH9ULeks5AUhKyJYztrPJc1UJL
diff --git a/tests/vo_metadata.xml b/tests/vo_metadata.xml
index 4dff0a76..3a61aa82 100644
--- a/tests/vo_metadata.xml
+++ b/tests/vo_metadata.xml
@@ -1,7 +1,7 @@
 <?xml version='1.0' encoding='UTF-8'?>
 <ns0:EntitiesDescriptor
   name="urn:mace:example.com:votest"
-  validUntil="2999-11-28T09:10:09Z"
+  validUntil="2036-11-28T09:10:09Z"
   xmlns:ns0="urn:oasis:names:tc:SAML:2.0:metadata">
   <ns0:EntityDescriptor
     entityID="urn:mace:example.com:it:tek">

---

For the short term I can merge this to current master - you can also apply it to your build to make the tests pass for that older version.
In the long term #518 should be rebased and merged.

[mcepl]

Thanks, works perfectly.

[c00kiemon5ter]

I am closing this and will it part of the next release. I am not running on a 32bit platform, and cannot guarantee that no such test will make it in master. If anything breaks again let me know. In the long term this issue should go away entirely by properly handling dates with the datetime module and not converting them.

---

Added by aaf6c54