TogoDX server dogfooding
TogoDX のバックエンドサーバ dx-server がアクセス多いと詰まるのでチューニングをする
code:sqlite
$ cd togodx-server/db
$ ln -s development.7.sqlite3 development.sqlite3
$ cd ..
$ docker run --rm -it -v $(pwd):/work -w /work -p 3000:3000 ruby:3.0.2 bash
# bundle install
# pumactl start
Puma starting in single mode...
* Puma version: 5.4.0 (ruby 3.0.2-p107) ("Super Flight")
* Min threads: 5
* Max threads: 5
* Environment: development
* PID: 860
Use Ctrl-C to stop
--- From another terminal window ---
# curl localhost:3000/breakdown/gene_chromosome_ensembl
{"label":"chr1","count":5477,"categoryId":"01","hasChild":false},{"label":"chr2","count":4197,"categoryId":"02","hasChild":false},{"label":"chr11","count":3363,"categoryId":"11","hasChild":false},{"label":"chr3","count":3188,"categoryId":"03","hasChild":false},{"label":"chr6","count":3065,"categoryId":"06","hasChild":false},{"label":"chr17","count":3057,"categoryId":"17","hasChild":false},{"label":"chr12","count":3057,"categoryId":"12","hasChild":false},{"label":"chr7","count":3017,"categoryId":"07","hasChild":false},{"label":"chr5","count":2988,"categoryId":"05","hasChild":false},{"label":"chr19","count":2988,"categoryId":"19","hasChild":false},{"label":"chr4","count":2658,"categoryId":"04","hasChild":false},{"label":"chr16","count":2560,"categoryId":"16","hasChild":false},{"label":"chr8","count":2483,"categoryId":"08","hasChild":false},{"label":"X","count":2425,"categoryId":"23","hasChild":false},{"label":"chr10","count":2332,"categoryId":"10","hasChild":false},{"label":"chr9","count":2330,"categoryId":"09","hasChild":false},{"label":"chr14","count":2287,"categoryId":"14","hasChild":false},{"label":"chr15","count":2222,"categoryId":"15","hasChild":false},{"label":"chr20","count":1459,"categoryId":"20","hasChild":false},{"label":"chr13","count":1402,"categoryId":"13","hasChild":false},{"label":"chr22","count":1386,"categoryId":"22","hasChild":false},{"label":"chr18","count":1242,"categoryId":"18","hasChild":false},{"label":"chr21","count":875,"categoryId":"21","hasChild":false},{"label":"Y","count":521,"categoryId":"24","hasChild":false},{"label":"MT","count":37,"categoryId":"25","hasChild":false}% ---
$ pumactl start -e production
Puma starting in single mode...
* Puma version: 5.4.0 (ruby 3.0.2-p107) ("Super Flight")
* Min threads: 5
* Max threads: 5
* Environment: production
* PID: 888
Use Ctrl-C to stop
---
$ rails s -e production
=> Booting Puma
=> Rails 6.1.4.1 application starting in production
=> Run bin/rails server --help for more startup options
Puma starting in single mode...
* Puma version: 5.4.0 (ruby 3.0.2-p107) ("Super Flight")
* Min threads: 5
* Max threads: 5
* Environment: production
* PID: 908
---
$ mkdir config/puma
$ cp config/puma.rb config/puma/production.rb
uncomment the worker line:
workers ENV.fetch("WEB_CONCURRENCY") { 2 })
$ rails s -e production
=> Booting Puma
=> Rails 6.1.4.1 application starting in production
=> Run bin/rails server --help for more startup options
926 Puma starting in cluster mode... 926 * Puma version: 5.4.0 (ruby 3.0.2-p107) ("Super Flight") 926 * Environment: production 926 * Restarts: (✔) hot (✔) phased 926 - Worker 0 (PID: 935) booted in 0.0s, phase: 0 926 - Worker 1 (PID: 942) booted in 0.0s, phase: 0 ---
$ pumactl start -e production
960 Puma starting in cluster mode... 960 * Puma version: 5.4.0 (ruby 3.0.2-p107) ("Super Flight") 960 * Environment: production 960 * Restarts: (✔) hot (✔) phased 962 + Gemfile in context: /work/Gemfile 964 + Gemfile in context: /work/Gemfile 960 - Worker 0 (PID: 962) booted in 2.26s, phase: 0 960 - Worker 1 (PID: 964) booted in 2.27s, phase: 0 動いた
何も設定しないと dev/prod 両方で Puma は single mode で起動する
config/puma/production.rb で workers の数を指定してあげるとクラスタモードで起動する
pumactl 使えということなんで pumactl でやっていきたい
PostgresDB 版を試す
docker-compose がある、ありがとう佐藤さん……
code:postgres
$ cd data
$ tar xf pg_data_idx.tar.gz
$ mv postgresql pg_data
$ docker compose -f docker-compose.prod.yml up
⠿ Container togodx-server-db-1 Recreated 0.1s
⠿ Container togodx-server-app-1 Recreated 0.1s
⠿ Container togodx-server-nginx-1 Recreated 0.1s
Attaching to togodx-server-app-1, togodx-server-db-1, togodx-server-nginx-1
togodx-server-app-1 | Using rake 13.0.6
togodx-server-app-1 | Using concurrent-ruby 1.1.9
togodx-server-app-1 | Using i18n 1.8.10
togodx-server-app-1 | Using minitest 5.14.4
togodx-server-app-1 | Using tzinfo 2.0.4
togodx-server-app-1 | Using zeitwerk 2.4.2
togodx-server-app-1 | Using activesupport 6.1.4.1
togodx-server-app-1 | Using builder 3.2.4
togodx-server-app-1 | Using erubi 1.10.0
togodx-server-app-1 | Using racc 1.5.2
togodx-server-app-1 | Using nokogiri 1.12.4 (x86_64-linux)
togodx-server-app-1 | Using rails-dom-testing 2.0.3
togodx-server-app-1 | Using crass 1.0.6
togodx-server-app-1 | Using loofah 2.12.0
togodx-server-app-1 | Using rails-html-sanitizer 1.4.2
togodx-server-app-1 | Using actionview 6.1.4.1
togodx-server-app-1 | Using rack 2.2.3
togodx-server-app-1 | Using rack-test 1.1.0
togodx-server-app-1 | Using actionpack 6.1.4.1
togodx-server-app-1 | Using nio4r 2.5.8
togodx-server-app-1 | Using websocket-extensions 0.1.5
togodx-server-app-1 | Using websocket-driver 0.7.5
togodx-server-app-1 | Using actioncable 6.1.4.1
togodx-server-app-1 | Using globalid 0.5.2
togodx-server-app-1 | Using activejob 6.1.4.1
togodx-server-app-1 | Using activemodel 6.1.4.1
togodx-server-app-1 | Using activerecord 6.1.4.1
togodx-server-app-1 | Using marcel 1.0.1
togodx-server-app-1 | Using mini_mime 1.1.1
togodx-server-app-1 | Using activestorage 6.1.4.1
togodx-server-app-1 | Using mail 2.7.1
togodx-server-app-1 | Using actionmailbox 6.1.4.1
togodx-server-app-1 | Using actionmailer 6.1.4.1
togodx-server-app-1 | Using actiontext 6.1.4.1
togodx-server-app-1 | Using active_interaction 4.0.5
togodx-server-app-1 | Using activerecord-import 1.2.0
togodx-server-app-1 | Using awesome_nested_set 3.4.0
togodx-server-app-1 | Using coderay 1.1.3
togodx-server-app-1 | Using better_errors 2.9.1
togodx-server-app-1 | Using debug_inspector 1.1.0
togodx-server-app-1 | Using binding_of_caller 1.0.0
togodx-server-app-1 | Using msgpack 1.4.2
togodx-server-app-1 | Using bootsnap 1.8.1
togodx-server-app-1 | Using bundler 2.2.22
togodx-server-app-1 | Using byebug 11.1.3
togodx-server-app-1 | Using connection_pool 2.2.5
togodx-server-app-1 | Using diff-lcs 1.4.4 28/588 togodx-server-app-1 | Using dotenv 2.7.6
togodx-server-app-1 | Using method_source 1.0.0
togodx-server-app-1 | Using thor 1.1.0
togodx-server-app-1 | Using railties 6.1.4.1
togodx-server-app-1 | Using dotenv-rails 2.7.6
togodx-server-app-1 | Using faraday-em_http 1.0.0
togodx-server-app-1 | Using faraday-em_synchrony 1.0.0
togodx-server-app-1 | Using faraday-excon 1.1.0
togodx-server-app-1 | Using faraday-httpclient 1.0.1
togodx-server-app-1 | Using faraday-net_http 1.0.1
togodx-server-app-1 | Using faraday-net_http_persistent 1.2.0
togodx-server-app-1 | Using faraday-patron 1.0.0
togodx-server-app-1 | Using faraday-rack 1.0.0
togodx-server-app-1 | Using multipart-post 2.1.1
togodx-server-app-1 | Using ruby2_keywords 0.0.5
togodx-server-app-1 | Using faraday 1.7.1
togodx-server-app-1 | Using ffi 1.15.4
togodx-server-app-1 | Using jbuilder 2.11.2
togodx-server-app-1 | Using rb-fsevent 0.11.0
togodx-server-app-1 | Using rb-inotify 0.10.1
togodx-server-app-1 | Using listen 3.7.0
togodx-server-app-1 | Using net-http-persistent 4.0.1
togodx-server-app-1 | Using pg 1.2.3
togodx-server-app-1 | Using pry 0.13.1
togodx-server-app-1 | Using pry-byebug 3.9.0
togodx-server-app-1 | Using pry-rails 0.3.9
togodx-server-app-1 | Using puma 5.4.0
togodx-server-app-1 | Using rack-cors 1.1.1
togodx-server-app-1 | Using sprockets 4.0.2
togodx-server-app-1 | Using sprockets-rails 3.2.2
togodx-server-app-1 | Using rails 6.1.4.1
togodx-server-app-1 | Using rails_pretty_json_rednerer 0.1.0
togodx-server-app-1 | Using rspec-support 3.10.2
togodx-server-app-1 | Using rspec-core 3.10.1
togodx-server-app-1 | Using rspec-expectations 3.10.1
togodx-server-app-1 | Using rspec-mocks 3.10.2
togodx-server-app-1 | Using rspec-rails 5.0.2
togodx-server-app-1 | Using rubystats 0.3.0
togodx-server-app-1 | Using sqlite3 1.4.2
togodx-server-app-1 | Bundle complete! 22 Gemfile dependencies, 86 gems now installed.
togodx-server-app-1 | Use bundle info [gemname] to see where a bundled gem is installed.
godx-server-nginx-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
togodx-server-nginx-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
togodx-server-nginx-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
togodx-server-nginx-1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
togodx-server-nginx-1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
togodx-server-nginx-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
togodx-server-nginx-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
togodx-server-nginx-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: using the "epoll" event method togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: nginx/1.21.4 togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: OS: Linux 5.10.47-linuxkit togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: start worker processes togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: start worker process 32 togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: start worker process 33 togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: start worker process 34 togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: start worker process 35 togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: start worker process 36 togodx-server-nginx-1 | 2021/11/22 02:37:19 notice 1#1: start worker process 37 togodx-server-db-1 |
togodx-server-db-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
togodx-server-db-1 |
togodx-server-db-1 | 2021-11-22 02:37:26.841 GMT 1 LOG: starting PostgreSQL 14.1 (Debian 14.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit togodx-server-db-1 | 2021-11-22 02:37:26.841 GMT 1 LOG: listening on IPv4 address "127.0.0.1", port 27500 togodx-server-db-1 | 2021-11-22 02:37:26.841 GMT 1 LOG: could not bind IPv6 address "::1": Cannot assign requested address togodx-server-db-1 | 2021-11-22 02:37:26.845 GMT 1 LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.27500" togodx-server-db-1 | 2021-11-22 02:37:27.019 GMT 29 LOG: database system was interrupted; last known up at 2021-10-16 06:20:37 GMT togodx-server-app-1 | => Booting Puma
togodx-server-app-1 | => Rails 6.1.4.1 application starting in production
togodx-server-app-1 | => Run bin/rails server --help for more startup options
togodx-server-db-1 | 2021-11-22 02:37:31.926 GMT 29 LOG: database system was not properly shut down; automatic recovery in progress togodx-server-db-1 | 2021-11-22 02:37:32.027 GMT 29 LOG: redo starts at 57/10BDD768 togodx-server-db-1 | 2021-11-22 02:37:32.027 GMT 29 LOG: invalid record length at 57/10BDD850: wanted 24, got 0 togodx-server-db-1 | 2021-11-22 02:37:32.027 GMT 29 LOG: redo done at 57/10BDD818 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s togodx-server-db-1 | 2021-11-22 02:37:32.085 GMT 1 LOG: database system is ready to accept connection togodx-server-app-1 | 1 Puma starting in cluster mode... togodx-server-app-1 | 1 * Puma version: 5.4.0 (ruby 3.0.2-p107) ("Super Flight") togodx-server-app-1 | 1 * Min threads: 10 togodx-server-app-1 | 1 * Max threads: 10 togodx-server-app-1 | 1 * Environment: production togodx-server-app-1 | 1 * Master PID: 1 togodx-server-app-1 | 1 * Workers: 10 togodx-server-app-1 | 1 * Restarts: (✔) hot (✖) phased togodx-server-app-1 | 1 * Preloading application togodx-server-app-1 | 1 Use Ctrl-C to stop togodx-server-app-1 | 1 - Worker 0 (PID: 19) booted in 0.02s, phase: 0 togodx-server-app-1 | 1 - Worker 1 (PID: 24) booted in 0.01s, phase: 0 togodx-server-app-1 | 1 - Worker 3 (PID: 44) booted in 0.01s, phase: 0 togodx-server-app-1 | 1 - Worker 2 (PID: 34) booted in 0.01s, phase: 0 togodx-server-app-1 | 1 - Worker 4 (PID: 49) booted in 0.01s, phase: 0 togodx-server-app-1 | 1 - Worker 5 (PID: 68) booted in 0.01s, phase: 0 togodx-server-app-1 | 1 - Worker 6 (PID: 75) booted in 0.01s, phase: 0 togodx-server-app-1 | 1 - Worker 7 (PID: 86) booted in 0.01s, phase: 0 togodx-server-app-1 | 1 - Worker 9 (PID: 107) booted in 0.0s, phase: 0 togodx-server-app-1 | 1 - Worker 8 (PID: 95) booted in 0.0s, phase: 0 ---
立ち上がったが curl のリクエストが通らない
500 (table "attribute" not found) が返ってくる
データがちゃんと読めていないっぽい
Databaseの設定をちゃんとしていないからでは
取ってきた data/pg_data/postgresql.conf を読んでいるのでこれを編集
code:data/pg_data/postgresql.conf
listen_addresses = '*' # what IP address(es) to listen on;
これでコンテナの外から db に繋がるようになった
次は config/database.yml を書き直す
code:config/database.yml
default: &default
adapter: postgresql
pool: <%= ENV.fetch('RAILS_MAX_THREADS') { 5 } %>
timeout: 5000
production:
<<: *default
database: togodx_server_development
port: 27500
wiki の記載を頼りにこのように修正してみたが compose up 時にエラーが出て database に接続できないと言われる
.env の内容を突っ込んでみる
code:config/database.yml
production:
<<: *default
database: togodx_server_development
port: 27500
ところが
FATAL: password authentication failed for user "togodx" (PG::ConnectionBad)
持ってきたdbファイルなのでパスワードが違うのかしら
psqlにはパスワードなしで入れるみたいなのでパスワードを設定します
code:psql
$ psql -h localhost -p 27500 -d togodx_server_development
psql (14.1 (Debian 14.1-1.pgdg110+1))
Type "help" for help.
togodx_server_development=# \du
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
togodx | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
togodx_server_development=# ALTER USER togodx WITH PASSWORD 'new_password';
ALTER ROLE
togodx_server_development=# \q
ここで設定した password を .env の TOGODX_SERVER_DATABASE_PASSWORD に書き込む
動きました
code:shell
$ curl localhost:8080/breakdown/gene_chromosome_ensembl
{"label":"chr1","count":5477,"categoryId":"01","hasChild":false},{"label":"chr2","count":4197,"categoryId":"02","hasChild":false},{"label":"chr11","count":3363,"categoryId":"11","hasChild":false},{"label":"chr3","count":3188,"categoryId":"03","hasChild":false},{"label":"chr6","count":3065,"categoryId":"06","hasChild":false},{"label":"chr17","count":3057,"categoryId":"17","hasChild":false},{"label":"chr12","count":3057,"categoryId":"12","hasChild":false},{"label":"chr7","count":3017,"categoryId":"07","hasChild":false},{"label":"chr5","count":2988,"categoryId":"05","hasChild":false},{"label":"chr19","count":2988,"categoryId":"19","hasChild":false},{"label":"chr4","count":2658,"categoryId":"04","hasChild":false},{"label":"chr16","count":2560,"categoryId":"16","hasChild":false},{"label":"chr8","count":2483,"categoryId":"08","hasChild":false},{"label":"X","count":2425,"categoryId":"23","hasChild":false},{"label":"chr10","count":2332,"categoryId":"10","hasChild":false},{"label":"chr9","count":2330,"categoryId":"09","hasChild":false},{"label":"chr14","count":2287,"categoryId":"14","hasChild":false},{"label":"chr15","count":2222,"categoryId":"15","hasChild":false},{"label":"chr20","count":1459,"categoryId":"20","hasChild":false},{"label":"chr13","count":1402,"categoryId":"13","hasChild":false},{"label":"chr22","count":1386,"categoryId":"22","hasChild":false},{"label":"chr18","count":1242,"categoryId":"18","hasChild":false},{"label":"chr21","count":875,"categoryId":"21","hasChild":false},{"label":"Y","count":521,"categoryId":"24","hasChild":false},{"label":"MT","count":37,"categoryId":"25","hasChild":false}% 長かった……