ChIP-Atlas Sapporo-EA deployment
Env
AWS EC2
m7a.2xlarge
EBS volume 5000GB (gp3 general purpose SSD)
Ubuntu 24.04
Steps
Set up ~/.minio-env with MINIO_ACCESS_KEY and MINIO_SECRET_KEY
Download reference files by download-reference.sh
Test CWL pipeline
Setup venv and docker
Set up Sapporo
Launch Sapporo
Test run Sapporo WES
Rewrite ChIP-Atlas app for WES
code:download-reference.sh
$ bash download-references.sh
+ ! -e /home/ubuntu/minio-binaries/mc
+ curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /home/ubuntu/minio-binaries/mc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 26.6M 100 26.6M 0 0 12.5M 0 0:00:02 0:00:02 --:--:-- 12.5M
+ chmod +x /home/ubuntu/minio-binaries/mc
+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/ubuntu/minio-binaries/
+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/ubuntu/minio-binaries/
+ . /home/ubuntu/.minio-env
++ export MINIO_ACCESS_KEY=
++ MINIO_ACCESS_KEY=
++ export MINIO_SECRET_KEY=
++ MINIO_SECRET_KEY=
+ mc alias set chip-atlas-dbcls https://chip-atlas.dbcls.jp
mc: Configuration written to /home/ubuntu/.mc/config.json. Please update your access credentials.
mc: Successfully created /home/ubuntu/.mc/share.
mc: Initialized share uploads /home/ubuntu/.mc/share/uploads.json file.
mc: Initialized share downloads /home/ubuntu/.mc/share/downloads.json file.
Added chip-atlas-dbcls successfully.
+ mkdir -p /home/ubuntu/chip-atlas/data/others/lib
+ mkdir -p /home/ubuntu/chip-atlas/data/metadata
+ mc cp --recursive chip-atlas-dbcls/data/others/lib/genome_size /home/ubuntu/chip-atlas/data/others/lib/genome_size
...r3.chrom.sizes: 111.63 KiB / 111.63 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 576.55 KiB/s 0s+ mc cp --recursive chip-atlas-dbcls/data/others/lib/id2symbol /home/ubuntu/chip-atlas/data/others/lib/id2symbol
...ol.sacCer3.tsv: 44.89 MiB / 44.89 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 75.06 MiB/s 0s+ mc cp --recursive chip-atlas-dbcls/data/others/lib/TSS /home/ubuntu/chip-atlas/data/others/lib/TSS
...SS.sacCer3.bed: 19.78 MiB / 19.78 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 18.53 MiB/s 1s+ mc cp --recursive chip-atlas-dbcls/data/metadata/experimentList.tab /home/ubuntu/chip-atlas/data/metadata/experimentList.tab
...rimentList.tab: 326.89 MiB / 326.89 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 52.63 MiB/s 6s+ mc cp --recursive chip-atlas-dbcls/data/metadata/fileList.tab /home/ubuntu/chip-atlas/data/metadata/fileList.tab
...a/fileList.tab: 129.30 MiB / 129.30 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 51.29 MiB/s 2s+ mc cp --recursive chip-atlas-dbcls/data/others/lib/inSilicoChIP /home/ubuntu/chip-atlas/data/others/lib/inSilicoChIP
...AllCell.bed.aa: 2.89 TiB / 2.89 TiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 124.63 MiB/s 6h44m39su
Set up venv
reference
Install cwltool
code:bash
$ sudo apt install python3-venv
$ cd
$ python3 -m venv ea-venv
$ source ~/ea-venv/bin/activate
$ pip3 install cwltool
$ cwltool --version
/home/ubuntu/ea-env/bin/cwltool 3.1.20241112140730
Install docker according to official document
edit job.yml to point the reference data
code:bash
$ cd chip-atlas/script/enrichment-analysis
$ cwltool --debug --validate enrichment-analysis.cwl job.yml
...
enrichment-analysis.cwl is valid CWL. No errors detected in the inputs.
$ cwltool --debug enrichment-analysis.cwl job.yml
...
{
"output_dir": {
"location": "file:///home/ubuntu/chip-atlas/script/enrichment-analysis/out",
"basename": "out",
"class": "Directory",
"listing": [
{
"class": "File",
"location": "file:///home/ubuntu/chip-atlas/script/enrichment-analysis/out/enrichment-analysis.result.tsv",
"basename": "enrichment-analysis.result.tsv",
"size": 12711,
"checksum": "sha1$518553189ca8c1a202289b358181924c03bbe0a4",
"path": "/home/ubuntu/chip-atlas/script/enrichment-analysis/out/enrichment-analysis.result.tsv"
},
{
"class": "File",
"location": "file:///home/ubuntu/chip-atlas/script/enrichment-analysis/out/enrichment-analysis.result.html",
"basename": "enrichment-analysis.result.html",
"size": 63456,
"checksum": "sha1$b21726969eb812c5e024bb2087b7d667bde19ac0",
"path": "/home/ubuntu/chip-atlas/script/enrichment-analysis/out/enrichment-analysis.result.html"
}
],
"path": "/home/ubuntu/chip-atlas/script/enrichment-analysis/out"
}
}INFO Final process status is success
Edit docker-compose.yml
code:docker-compose.yml
services:
sapporo:
image: ghcr.io/sapporo-wes/sapporo-service:2.0.4
container_name: sapporo-enrichment-analysis
volumes:
- ${PWD}/sapporo_config:/app/sapporo_config
- /home/ubuntu/chip-atlas/script/enrichment-analysis:/chip-atlas/enrichment-analysis
- /home/ubuntu/chip-atlas/data:/chip-atlas/data
- /var/run/docker.sock:/var/run/docker.sock
environment:
- SAPPORO_DEBUG=True
- SAPPORO_SERVICE_INFO=/app/sapporo_config/service-info.json
- SAPPORO_RUN_SH=/app/sapporo_config/run.sh
- ENRICHMENT_ANALYSIS_DIR=/chip-atlas/enrichment-analysis
- ENRICHMENT_ANALYSIS_DATA_DIR=/chip-atlas/data
restart: on-failure
working_dir: /app
network_mode: host
command: "sapporo"
ports:
- "1122:1122"
code:bash
$ docker compose up
$ curl -s localhost:1122/service-info | jq .
{
"id": "sapporo-service",
"name": "sapporo-service",
"type": {
"group": "sapporo-wes",
"artifact": "wes",
"version": "sapporo-wes-2.0.0"
},
"description": "The instance of the Sapporo-WES.",
"organization": {
"name": "Sapporo-WES Project Team",
"url": "https://github.com/orgs/sapporo-wes/people"
},
"contactUrl": "https://github.com/sapporo-wes/sapporo-service/issues",
"documentationUrl": "https://github.com/sapporo-wes/sapporo-service/blob/main/README.md",
"createdAt": "2024-12-11T02:27:06Z",
"updatedAt": "2024-12-11T02:27:06Z",
"environment": null,
"version": "2.0.4",
"workflow_type_versions": {
"enrichment-analysis": {
"workflow_type_version": [
"1.0.0"
]
}
},
"supported_wes_versions": [
"1.1.0",
"sapporo-wes-2.0.0"
],
"supported_filesystem_protocols": [
"http",
"https",
"file"
],
"workflow_engine_versions": {
"enrichment-analysis": {
"workflow_engine_version": [
"1.0.0"
]
}
},
"default_workflow_engine_parameters": {},
"system_state_counts": {
"UNKNOWN": 0,
"QUEUED": 0,
"INITIALIZING": 0,
"RUNNING": 0,
"PAUSED": 0,
"COMPLETE": 0,
"EXECUTOR_ERROR": 0,
"SYSTEM_ERROR": 0,
"CANCELED": 0,
"CANCELING": 0,
"PREEMPTED": 0,
"DELETED": 0,
"DELETING": 0
},
"auth_instructions_url": "https://github.com/sapporo-wes/sapporo-service/blob/main/README.md#authentication",
"tags": {}
}
run test like
code:bash
$ curl -s -X POST http://0.0.0.0:1122/runs | jq .
{
"msg": "{'type': 'missing', 'loc': ('body', 'workflow_type'), 'msg': 'Field required', 'input': None}, {'type': 'missing', 'loc': ('body', 'workflow_engine'), 'msg': 'Field required', 'input': None}",
"status_code": 400
}
$ curl -s -X POST -F "workflow_type=enrichment-analysis" -F "workflow_engine=enrichment-analysis" http://0.0.0.0:1122/runs | jq .
{
"run_id": "de0b6d73-ddc6-4878-9e94-32b18ef71166"
}
$ curl http://0.0.0.0:1122/runs/de0b6d73-ddc6-4878-9e94-32b18ef71166 | jq .
{
"run_id": "de0b6d73-ddc6-4878-9e94-32b18ef71166",
"request": {
"workflow_params": {},
"workflow_type": "enrichment-analysis",
"workflow_type_version": "1.0.0",
"tags": {},
"workflow_engine": "enrichment-analysis",
"workflow_engine_version": "1.0.0",
"workflow_engine_parameters": null,
"workflow_url": ""
},
"state": "EXECUTOR_ERROR",
"run_log": {
"name": null,
"cmd": [
"docker",
"run",
"--rm",
"-v",
"/var/run/docker.sock:/var/run/docker.sock",
"-e",
"DOCKER_HOST=unix:///var/run/docker.sock",
"-v",
"/tmp:/tmp",
"-v",
"/app/runs/de/de0b6d73-ddc6-4878-9e94-32b18ef71166:/app/runs/de/de0b6d73-ddc6-4878-9e94-32b18ef71166",
"-w=/app/runs/de/de0b6d73-ddc6-4878-9e94-32b18ef71166/exe",
"quay.io/commonwl/cwltool:3.1.20240508115724",
"--outdir",
"/app/runs/de/de0b6d73-ddc6-4878-9e94-32b18ef71166/outputs",
"/chip-atlas/enrichment-analysis/enrichment-analysis.cwl",
"/chip-atlas/enrichment-analysis/job.yml",
"1>/app/runs/de/de0b6d73-ddc6-4878-9e94-32b18ef71166/stdout.log",
"2>/app/runs/de/de0b6d73-ddc6-4878-9e94-32b18ef71166/stderr.log"
],
"start_time": "2024-12-11T02:35:04Z",
"end_time": "2024-12-11T02:35:23",
"stdout": "",
"stderr": "Unable to find image 'quay.io/commonwl/cwltool:3.1.20240508115724' locally\n3.1.20240508115724: Pulling from commonwl/cwltool\n1207c741d8c9: Pulling fs layer\na3c7bc1de874: Pulling fs layer\n18dfb1d5a53f: Pulling fs layer\n813fee53afe1: Pulling fs layer\nbc33fb833569: Pulling fs layer\nb06818a9d84f: Pulling fs layer\n8a9f357de77c: Pulling fs layer\nf55603fafb3b: Pulling fs layer\nd68f9dcffbaa: Pulling fs layer\nbc33fb833569: Waiting\nb06818a9d84f: Waiting\n8a9f357de77c: Waiting\nf55603fafb3b: Waiting\nd68f9dcffbaa: Waiting\n813fee53afe1: Waiting\n18dfb1d5a53f: Verifying Checksum\n18dfb1d5a53f: Download complete\na3c7bc1de874: Verifying Checksum\na3c7bc1de874: Download complete\n1207c741d8c9: Verifying Checksum\n1207c741d8c9: Download complete\n1207c741d8c9: Pull complete\na3c7bc1de874: Pull complete\n18dfb1d5a53f: Pull complete\n813fee53afe1: Verifying Checksum\n813fee53afe1: Download complete\n813fee53afe1: Pull complete\nbc33fb833569: Verifying Checksum\nbc33fb833569: Download complete\nbc33fb833569: Pull complete\nf55603fafb3b: Download complete\nb06818a9d84f: Verifying Checksum\nb06818a9d84f: Download complete\n8a9f357de77c: Verifying Checksum\n8a9f357de77c: Download complete\nb06818a9d84f: Pull complete\nd68f9dcffbaa: Download complete\n8a9f357de77c: Pull complete\nf55603fafb3b: Pull complete\nd68f9dcffbaa: Pull complete\nDigest: sha256:6ea8395d50e20a35aa3814b35bb54307e141fb406b2e0546328830d3d0bf6de8\nStatus: Downloaded newer image for quay.io/commonwl/cwltool:3.1.20240508115724\n\u001b[1;30mINFO\u001b[0m /usr/local/bin/cwltool 3.1.20240508115724\nTraceback (most recent call last):\n File \"/usr/local/bin/cwltool\", line 8, in <module>\n sys.exit(run())\n ^^^^^\n File \"cwltool/main.py\", line 1470, in run\n File \"cwltool/main.py\", line 1074, in main\n File \"cwltool/load_tool.py\", line 108, in resolve_tool_uri\nschema_salad.exceptions.ValidationException: Not found: '/chip-atlas/enrichment-analysis/enrichment-analysis.cwl'",
"exit_code": 1,
"system_logs": []
},
"task_logs_url": null,
"task_logs": null,
"outputs": null
}
Only workflow_type and workflow_engine were required to invoke a run
fix job params