pytest：如何仅抑制一个测试用例的回溯

Question

我有以下测试用例：

    @pytest.mark.parametrize("init_resp", [True, False])
    @pytest.mark.parametrize("mechanism", ["login", "plain"])
    def test_byclient(self, auth_peeker_controller, client, mechanism, init_resp):
        self._ehlo(client)
        client.user = "goodlogin"
        client.password = "goodpasswd"
        auth_meth = getattr(client, "auth_" + mechanism)
        try:
            client.auth(mechanism, auth_meth, initial_response_ok=init_resp)
        except SMTPAuthenticationError:
            if (mechanism, init_resp) == ("login", False):
                client.docmd("*")
                pytest.xfail(reason="smtplib.SMTP.auth_login is buggy (bpo-27820)")
            else:
                raise
        peeker = auth_peeker_controller.handler
        assert isinstance(peeker, PeekerHandler)
        assert peeker.login == b"goodlogin"
        assert peeker.password == b"goodpasswd"

如您所见，对于一个参数组合（即("login", False)），测试预计会失败

问题是，我仍然有两个异常扰乱了pytest的输出：

aiosmtpd\tests\test_smtp.py:1054: in test_byclient
    client.auth(mechanism, auth_meth, initial_response_ok=init_resp)
C:\Python\36\lib\smtplib.py:642: in auth
    raise SMTPAuthenticationError(code, resp)
E   smtplib.SMTPAuthenticationError: (334, b'UGFzc3dvcmQA')

During handling of the above exception, another exception occurred:
aiosmtpd\tests\test_smtp.py:1058: in test_byclient
    pytest.xfail(reason="smtplib.SMTP.auth_login is buggy (bpo-27820)")
E   _pytest.outcomes.XFailed: smtplib.SMTP.auth_login is buggy (bpo-27820)

（pytest使用--tb=short运行）

是否可以仅针对这一个测试用例抑制这些异常

---

编辑1

实际上，在发布这个问题之后，我将测试用例更改为：

    @pytest.mark.parametrize("init_resp", [True, False])
    @pytest.mark.parametrize("mechanism", ["login", "plain"])
    def test_byclient(self, auth_peeker_controller, client, mechanism, init_resp):
        self._ehlo(client)
        client.user = "goodlogin"
        client.password = "goodpasswd"
        auth_meth = getattr(client, "auth_" + mechanism)
        if (mechanism, init_resp) == ("login", False):
            with pytest.raises(SMTPAuthenticationError):
                client.auth(mechanism, auth_meth, initial_response_ok=init_resp)
            client.docmd("*")
            pytest.xfail(reason="smtplib.SMTP.auth_login is buggy (bpo-27820)")
        client.auth(mechanism, auth_meth, initial_response_ok=init_resp)
        peeker = auth_peeker_controller.handler
        assert isinstance(peeker, PeekerHandler)
        assert peeker.login == b"goodlogin"
        assert peeker.password == b"goodpasswd"

结果是有点更好：

aiosmtpd\tests\test_smtp.py:1035: in test_byclient
    pytest.xfail(reason="smtplib.SMTP.auth_login is buggy (bpo-27820)")
E   _pytest.outcomes.XFailed: smtplib.SMTP.auth_login is buggy (bpo-27820)

不过，如果可能的话，我仍然希望这个“例外”被抑制

---

上面的代码片段来自these lines on GitHub

Answer 1

如果您不介意显式地写出要测试的每个参数组合，那么很容易将其中一个标记为预期失败。我个人更喜欢这种风格，因为我觉得它更清晰、更灵活：

@pytest.mark.parametrize(
    'init_resp,mechanism', [
        pytest.param(True,  'login'),
        pytest.param(False, 'login', marks=pytest.xfail),
        pytest.param(True,  'plain'),
        pytest.param(False, 'plain'),
    ]
)
def test_byclient(self, auth_peeker_controller, client, mechanism, init_resp):
    ...

也就是说，在我看来，您给出的示例没有输出任何异常。它只是说3个测试通过，1个测试失败。我怀疑您的问题与pytest的配置有关，但如果您能发布一个简单的工作示例，这将非常有用

Answer 2

您的测试用例不应该像您的PROD代码一样try except异常。 pytest提供了一种机制来预期异常，然后处理它。下面是代码

with pytest.raises(SMTPAuthenticationError) as exception:
        client.auth(mechanism, auth_meth, initial_response_ok=init_resp)
        .... do something.

Answer 3

有几种方法可以将xfail标记附加到测试函数之外。例如，在测试集合完成时：

import pytest


def pytest_collection_modifyitems(items):
    for item in items:
        if item.name == "test_byclient[login-False]":
            item.add_marker(pytest.mark.xfail(reason="..."))

（将代码放在测试rootdir的conftest.py中）

或通过autouse夹具：

@pytest.fixture(autouse=True)
def _(request):
    if request.node.name == "test_byclient[login-False]":
        request.node.add_marker(pytest.mark.xfail(reason="..."))

这样，特定于login-False的代码可以一起从测试函数中删除：

@pytest.mark.parametrize("init_resp", [True, False])
@pytest.mark.parametrize("mechanism", ["login", "plain"])
def test_byclient(self, auth_peeker_controller, client, mechanism, init_resp):
    self._ehlo(client)
    client.user = "goodlogin"
    client.password = "goodpasswd"
    auth_meth = getattr(client, "auth_" + mechanism)
    client.auth(mechanism, auth_meth, initial_response_ok=init_resp)
    peeker = auth_peeker_controller.handler
    assert isinstance(peeker, PeekerHandler)
    assert peeker.login == b"goodlogin"
    assert peeker.password == b"goodpasswd"

如果client.docmd("*")是login-False-测试的一个组成部分，那么我同意@jornsharpe的观点：您有一个测试函数覆盖两个不同的测试用例，应该将其拆分为两个单独的测试用例