Schemathesis
WebAPIのサーバ実装をProperty-Based Testingで検証するコマンドラインのTesting Tool
HypothesisをベースにSwagger/OpenAPI/GraphQLに対応している
ユニットテストフレームワークへの組み込みにも対応している
@c_bata_ から教えてもらった https://twitter.com/c_bata_/status/1260776537969717248
https://github.com/kiwicom/schemathesis
Schemathesis is a tool for testing your web applications built with Open API / Swagger or GraphQL specifications. It reads the application schema and generates test cases which will ensure that your application is compliant with its schema. The application under test could be written in any language, the only thing you need is a valid API schema in a supported format.
(DeepL訳) Schemathesisは、Open API / SwaggerやGraphQL仕様で構築されたWebアプリケーションをテストするためのツールです。アプリケーションのスキーマを読み取り、アプリケーションがスキーマに準拠していることを確認するテストケースを生成します。テスト対象のアプリケーションはどの言語でも書くことができ、必要なのはサポートされている形式の有効なAPIスキーマだけです。
Supported specification versions:
Swagger 2.0
Open API 3.0.x
GraphQL June 2018
やってみた
2020/09/26
https://github.com/schemathesis/schemathesis/blob/master/README.rst を読む
まず環境を作る
code:shell
$ python3 -m venv venv-wsl
$ source venv-wsl/bin/activate
(venv)$ pip install schemathesis
(venv)$ pip list
Package Version
--------------------- ---------
attrs 19.3.0
certifi 2020.6.20
chardet 3.0.4
click 7.1.2
graphql-core 3.1.2
hypothesis 5.36.1
hypothesis-graphql 0.3.1
hypothesis-jsonschema 0.18.0
idna 2.10
importlib-metadata 1.7.0
iniconfig 1.0.1
jsonschema 3.2.0
junit-xml 1.9
more-itertools 8.5.0
packaging 20.4
pip 20.2.3
pkg-resources 0.0.0
pluggy 0.13.1
py 1.9.0
pyparsing 2.4.7
pyrsistent 0.17.3
pytest 6.0.2
pytest-subtests 0.3.2
PyYAML 5.3.1
requests 2.24.0
schemathesis 2.4.1
setuptools 39.0.1
six 1.15.0
sortedcontainers 2.2.2
starlette 0.13.8
toml 0.10.1
urllib3 1.25.10
Werkzeug 1.0.1
wheel 0.35.1
zipp 3.2.0
次は以下のようなコマンドを実行するため、 OpenAPIドキュメントが必要
schemathesis run https://example.com/api/swagger.json
FastAPIで立ち上げておく
code:main.py
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class Message(BaseModel):
message: str
@app.get("/hello", response_model=Message)
async def root(name: str, age: int):
return {"message": f"Hello {name}({age})"}
code:shell
(venv2)$ uvicorn main:app --reload
INFO: Started server process 4080
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
cURL でOpenAPIドキュメントを取得してみる
code:openapi.json
$ curl -s -X GET http://localhost:8000/openapi.json | python -m json.tool
{
"openapi": "3.0.2",
"info": {
"title": "FastAPI",
"version": "0.1.0"
},
"paths": {
"/hello": {
"get": {
"summary": "Root",
"operationId": "root_hello_get",
"parameters": [
{
"required": true,
"schema": {
"title": "Name",
"type": "string"
},
"name": "name",
"in": "query"
},
{
"required": true,
"schema": {
"title": "Age",
"type": "integer"
},
"name": "age",
"in": "query"
}
],
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Message"
}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"HTTPValidationError": {
"title": "HTTPValidationError",
"type": "object",
"properties": {
"detail": {
"title": "Detail",
"type": "array",
"items": {
"$ref": "#/components/schemas/ValidationError"
}
}
}
},
"Message": {
"title": "Message",
"required": [
"message"
],
"type": "object",
"properties": {
"message": {
"title": "Message",
"type": "string"
}
}
},
"ValidationError": {
"title": "ValidationError",
"required": [
"loc",
"msg",
"type"
],
"type": "object",
"properties": {
"loc": {
"title": "Location",
"type": "array",
"items": {
"type": "string"
}
},
"msg": {
"title": "Message",
"type": "string"
},
"type": {
"title": "Error Type",
"type": "string"
}
}
}
}
}
}
SchemathesisをWindowsで実行すると Unreliable test timings! と言われ失敗するため、Linux環境から実行しなおし
Schemathesisを実行、1つのAPIに100回テストが実行された
https://scrapbox.io/files/5f6eda79479cb7001edcac65.png
code:shell
$ schemathesis run http://localhost:8000/openapi.json
=============================== Schemathesis test session starts ===============================platform Linux -- Python 3.6.8, schemathesis-2.4.1, hypothesis-5.36.1, hypothesis_jsonschema-0.18.0, jsonschema-3.2.0
rootdir: /mnt/c/Project/python3/schemathesis
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/mnt/c/Project/python3/schemathesis/.hypothesis/examples')
Schema location: http://localhost:8000/openapi.json
Base URL: http://localhost:8000/
Specification version: Open API 3.0.2
Workers: 1
collected endpoints: 1
GET /hello . 100%
=========================================== SUMMARY ============================================
Performed checks:
not_a_server_error 100 / 100 passed PASSED
====================================== 1 passed in 1.45s =======================================
Schemathesis実行中のAPIサーバー側ログ
https://scrapbox.io/files/5f6edafd9c66b10024cf5d45.png
code:console.log
INFO: Started server process 28928
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 127.0.0.1:51076 - "GET /openapi.json HTTP/1.1" 200 OK
INFO: 127.0.0.1:51078 - "GET /openapi.json HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name=0 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=15464&name=%1A%C3%9A%C3%BD7%C2%9D%C2%85 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-21497&name=%05%C3%BA%F3%BF%8E%98m HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=684&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-17770&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=20665&name=%F2%AC%B8%B9 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-2718&name=zf%F3%81%A7%BA%C2%97 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=61&name=f%F3%81%A7%BA%C2%97 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-27745&name=%05%C2%B8 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=384&name=%F1%80%A6%B8 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=384&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=384&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=384&name=%C3%A2%C2%A4%C2%B7%C3%BE HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=84&name=%C2%BB%F0%AD%A1%BD%13 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=84&name=%F3%B2%96%B5%2A%18%C3%9F%F1%B8%85%8C%19%5D%C3%A64%C2%81D%C2%A0%F2%89%90%9D%F2%82%B2%8D%C3%AB%C3%9A%C2%B7%C2%BF%0F%29%C3%8F%3F HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-1409163278&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-1409163278&name=%F2%80%95%806 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-1409163278&name=66 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-100&name=%F2%92%96%BC HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=0&name=%F0%91%AF%99%C2%BD%F2%92%96%BC HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=8263035874490490863&name= HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-66&name=%1A%F1%80%BD%A3Q%C3%8EG%C3%8D%C2%94%24%F3%B6%BD%9B%C2%8E%C2%B9-%C2%A5%C3%B6%2A5 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-66&name=%24%F1%80%BD%A3Q%C3%8EG%C3%8D%C2%94%24%F3%B6%BD%9B%C2%8E%C2%B9-%C2%A5%C3%B6%2A5 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-66&name=%24%F1%80%BD%A3Q%C3%8EG%C3%8D%C2%94%24%F3%B6%BD%9B%C2%8E%C2%B9-%C2%A5%C3%B6%2A5 HTTP/1.1" 200 OK
INFO: 127.0.0.1:51079 - "GET /hello?age=-4707&name=%3C%C3%9F%1A%C2%81 HTTP/1.1" 200 OK
WSGIアプリケーションを直接指定して実行も可能
code:shell
$ PYTHONPATH=. schemathesis run --app=main:app /openapi.json
pytestから実行する
Webサーバーは事前に立ち上げておく
code:tests.py
import requests
import schemathesis
schema = schemathesis.from_uri("http://0.0.0.0:8000/openapi.json")
@schema.parametrize()
def test_no_server_errors(case):
response = case.call()
case.validate_response(response)
assert response.status_code < 500
code:shell
$ pytest tests.py
===================================== test session starts ======================================
platform linux -- Python 3.6.8, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: /mnt/c/Project/python3/schemathesis
plugins: hypothesis-5.36.1, subtests-0.3.2, schemathesis-2.4.1
collected 1 item
tests.py . 100%
====================================== 1 passed in 1.98s =======================================
100テスト実行されていても、pytest的には1テストの扱いで表示される
https://schemathesis.readthedocs.io/en/stable/usage.html を読む
デバッグ用に、実行時の通信内容をVCRのcassette format形式でYAMLに保存できる
code:shell
$ schemathesis run http://localhost:8000/openapi.json --store-network-log cassette.yaml
code:cassette.yaml
command: 'schemathesis run http://localhost:8000/openapi.json --store-network-log cassette.yaml'
recorded_with: 'Schemathesis 2.4.1'
http_interactions:
- id: '1'
status: 'SUCCESS'
seed: '234690900641751568044503997757241782401'
elapsed: '0.003724'
recorded_at: '2020-09-26T15:11:30.358730'
request:
uri: 'http://localhost:8000/hello?age=0&name='
method: 'GET'
headers:
User-Agent:
- 'schemathesis/2.4.1'
Accept-Encoding:
- 'gzip, deflate'
Accept:
- '*/*'
Connection:
- 'keep-alive'
body:
encoding: 'utf-8'
base64_string: ''
response:
status:
code: '200'
message: 'OK'
headers:
date:
- 'Sat, 26 Sep 2020 06:11:30 GMT'
server:
- 'uvicorn'
content-length:
- '23'
content-type:
- 'application/json'
body:
encoding: 'utf8'
base64_string: 'eyJtZXNzYWdlIjoiSGVsbG8gKDApIn0='
http_version: '1.1'
- id: '2'
...
保存したcassette formatファイルを指定して再実行できる。失敗したものだけ再実行もできる
code:shell
(venv-wsl)$ schemathesis replay cassette.yaml --status=FAILURE
Replaying cassette: cassette.yaml
Total interactions: 100
(venv-wsl)$ schemathesis replay cassette.yaml
Replaying cassette: cassette.yaml
Total interactions: 100
ID : 1
URI : http://localhost:8000/hello?age=0&name=
Old status code : 200
New status code : 200
ID : 2
URI : http://localhost:8000/hello?age=0&name=0
Old status code : 200
New status code : 200
ID : 3
URI : http://localhost:8000/hello?age=0&name=
Old status code : 200
New status code : 200
ID : 4
URI : http://localhost:8000/hello?age=0&name=
Old status code : 200
New status code : 200
https://scrapbox.io/files/5f6ee705fac2ce001ed63aa1.png
JUnit形式の結果出力
code:shell
$ schemathesis run http://localhost:8000/openapi.json --junit-xml=junit.xml
code:junit.xml
<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="0" tests="1" time="1.2449706999996124">
<testsuite disabled="0" errors="0" failures="0" hostname="SHIMIZUKAWA-X1C2018" name="schemathesis" skipped="0" tests="1" time="1.2449706999996124">
<testcase name="GET /hello" time="1.244971"/>
</testsuite>
</testsuites>