Nginxで正規表現を使ってbotを弾く

ISUCON10の問題より

Nginxで正規表現を使ってbotを弾くテクニック

code:nginx.conf

map $http_user_agent $is_bot {

default 0;

"~*^www.domain.com:Agent.*$" 1;

"~*ISUCONbot(-Mobile)?" 1;

"~*ISUCONbot-Image\/" 1;

"~*Mediapartners-ISUCON" 1;

"~*ISUCONCoffee" 1;

"~*ISUCONFeedSeeker(Beta)?" 1;

"~*crawler \(https:\/\/isucon\.invalid\/(support\/faq\/|help\/jp\/)" 1;

"~*isubot" 1;

"~*Isupider" 1;

"~*Isupider(-image)?\+" 1;

"~*(bot|crawler|spider)(?:.\/;@()|$)/" 1;

}

server {

root /home/isucon/isucon10-qualify/webapp/public;

listen 80 default_server;

listen :::80 default_server;

if ($is_bot) {

return 503;

}

code:shell

root@team124-001:~# curl http://127.0.0.1:80/ -H 'User-Agent: isubot' -I

HTTP/1.1 503 Service Temporarily Unavailable

Server: nginx/1.14.0 (Ubuntu)

Date: Tue, 22 Sep 2020 14:45:14 GMT

Content-Type: text/html

Content-Length: 222

Connection: keep-alive