Saving BLAST Results to AWS S3
The pre-installed BLAST workflow template obtains query sequences with efetch from NCBI or curl from
any HTTPS URL and saves results to a configurable path under the persistent DataVolume. To fetch
query sequences from AWS S3, save results back to AWS S3, and drop support for custom BLAST
databases, first follow the workflow catalog
documentation to copy the BLAST workflow template and name
the copy BLAST S3. Then open the detail view on the copied template and make the changes below:
Because this copy drops some functionality, the template is simpler. It uses volume ingress and egress instead of a job that fetches data and writes it to a persistent volume. You can use the following template:
{{- $dbpath := list "/data" .BlastDbPath | join "/" }}
{{- $dbname := .BlastDbName }}
{{- $s3_uri := trim .S3Uri }}
{{- if not (hasPrefix "s3://" $s3_uri) }}
{{- fail "S3Uri is required and has to start with 's3://'" }}
{{- end }}
{{- $s3_secret := trim .S3Secret }}
{{- if not (hasPrefix "secret://" $s3_secret) }}
{{- fail "S3Secret is required and has to start with 'secret://'" }}
{{- end }}
{{- $run_name := trim .RunName }}
{{- if not $run_name }}
{{- fail "RunName is required" }}
{{- end }}
version: v4
volumes:
data:
use: {{.DataVolumeProvisioner}}
name: {{.DataVolumeName}}
scratch:
use: {{.ScratchVolumeProvisioner}}
size: 1GB
ingress:
- source:
uri: "{{$s3_uri}}/{{$run_name}}.fa"
secret: {{$s3_secret}}
destination:
uri: "file://{{$run_name}}.fa"
egress:
- source:
uri: "file://{{$run_name}}.blast.out"
destination:
uri: "{{$s3_uri}}/{{$run_name}}.blast.out"
secret: {{$s3_secret}}
jobs:
fetch-db:
image:
uri: {{.WorkflowContainer}}
mounts:
/data: data
/scratch: scratch
script: |
#!/bin/bash
mkdir -p "{{$dbpath}}" && cd "{{$dbpath}}" || exit 1
# fix update_blastdb.pl if it's from a conda container
ubdb="$(type -p update_blastdb.pl)"
curl="$(type -p curl)"
[[ -z "$ubdb" || -z "$curl" ]] && exit 1
if [[ "$ubdb" =~ conda ]]; then
sed "s:^my \\\$curl.*$:my \$curl = '$curl';:" "${ubdb}" > update_blastdb.pl
else
cp "${ubdb}" update_blastdb.pl
fi
chmod 750 update_blastdb.pl
if ./update_blastdb.pl --showall | grep -q {{$dbname}} ; then
echo "{{$dbname}} is a public database available from NCBI"
now=$(date +%s)
if [[ -e {{$dbname}}__ ]]; then
last=$(cat {{$dbname}}__)
if (( (now - last) < 86400 )) ; then
echo " {{$dbname}} is current - update skipped."
exit
fi
fi
echo $now > {{$dbname}}__
echo " updating/downloading {{$dbname}}"
./update_blastdb.pl --num_threads=2 --decompress {{$dbname}} && exit 0 || exit 1
else
echo "{{$dbname}} is not a public database"
exit 1
fi
resource:
cpu:
cores: 1
threads: true
memory:
size: 4GiB
policy:
timeout:
execute: {{.BlastFetchTimeout}}
run-blast:
image:
uri: {{.WorkflowContainer}}
mounts:
/data: data
/scratch: scratch
cwd: /scratch
script: |
#!/bin/bash
{{.BlastCmd}} -num_threads {{.BlastCores}} -query "{{$run_name}}.fa" -db {{$dbpath}}/{{$dbname}} -out {{$run_name}}.blast.out {{.BlastOpts}} || exit 1
cat {{$run_name}}.blast.out
resource:
cpu:
cores: {{.BlastCores}}
affinity: NUMA
memory:
size: {{.BlastMemory}}
policy:
timeout:
execute: {{.BlastQueryTimeout}}
depends-on:
- name: fetch-db
status: FINISHED
- Remove the following parameters:
CustomBlastDbFetchCmd,CustomBlastDbOptions,CustomBlastDbName,CustomBlastDbTimeout,RetrieveQuerySequencesCmd, andBlastOutputPath - Add two new parameters:
- Name:
S3UriType:TextDescription: AWS URI prefix for blast inputs and outputs. Default: A URI likes3://<bucket>/<path...> - Name:
S3SecretType:TextDescription: Secret from the secret store with AWS credentials. Default: A URI likesecret://<scope>/<name>
- Name:
Once you save the modified BLAST workflow template, you can run it the same way as the stock BLAST
workflow template, supplying the new S3Uri and S3Secret parameters.