Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Saving BLAST Results to AWS S3

The pre-installed BLAST workflow template obtains query sequences with efetch from NCBI or curl from any HTTPS URL and saves results to a configurable path under the persistent DataVolume. To fetch query sequences from AWS S3, save results back to AWS S3, and drop support for custom BLAST databases, first follow the workflow catalog documentation to copy the BLAST workflow template and name the copy BLAST S3. Then open the detail view on the copied template and make the changes below:

Replace the workflow template

Because this copy drops some functionality, the template is simpler. It uses volume ingress and egress instead of a job that fetches data and writes it to a persistent volume. You can use the following template:

{{- $dbpath := list "/data" .BlastDbPath | join "/"  }}
{{- $dbname := .BlastDbName }}

{{- $s3_uri := trim .S3Uri }}
{{- if not (hasPrefix "s3://" $s3_uri) }}
  {{- fail "S3Uri is required and has to start with 's3://'" }}
{{- end }}

{{- $s3_secret := trim .S3Secret }}
{{- if not (hasPrefix "secret://" $s3_secret) }}
  {{- fail "S3Secret is required and has to start with 'secret://'" }}
{{- end }}

{{- $run_name := trim .RunName }}
{{- if not $run_name }}
  {{- fail "RunName is required" }}
{{- end }}

version: v4
volumes:
  data:
    use: {{.DataVolumeProvisioner}}
    name: {{.DataVolumeName}}
  scratch:
    use: {{.ScratchVolumeProvisioner}}
    size: 1GB
    ingress:
      - source:
          uri: "{{$s3_uri}}/{{$run_name}}.fa"
          secret: {{$s3_secret}}
        destination:
          uri: "file://{{$run_name}}.fa"
    egress:
      - source:
          uri: "file://{{$run_name}}.blast.out"
        destination:
          uri: "{{$s3_uri}}/{{$run_name}}.blast.out"
          secret: {{$s3_secret}}
jobs:
  fetch-db:
    image:
      uri: {{.WorkflowContainer}}
    mounts:
      /data: data
      /scratch: scratch
    script: |
      #!/bin/bash
      mkdir -p "{{$dbpath}}" && cd "{{$dbpath}}" || exit 1
      # fix update_blastdb.pl if it's from a conda container
      ubdb="$(type -p update_blastdb.pl)"
      curl="$(type -p curl)"
      [[ -z "$ubdb" || -z "$curl" ]] && exit 1
      if [[ "$ubdb" =~ conda ]]; then
        sed "s:^my \\\$curl.*$:my \$curl = '$curl';:" "${ubdb}" > update_blastdb.pl
      else
        cp "${ubdb}"  update_blastdb.pl
      fi
      chmod 750 update_blastdb.pl

      if  ./update_blastdb.pl --showall | grep -q {{$dbname}} ; then
        echo "{{$dbname}} is a public database available from NCBI"
        now=$(date +%s)
        if [[ -e {{$dbname}}__ ]]; then
          last=$(cat {{$dbname}}__)
          if (( (now - last) < 86400 )) ; then
            echo "  {{$dbname}} is current - update skipped."
            exit
            fi
          fi
          echo $now > {{$dbname}}__
          echo "  updating/downloading {{$dbname}}"
          ./update_blastdb.pl --num_threads=2 --decompress {{$dbname}} && exit 0 || exit 1
        else
          echo "{{$dbname}} is not a public database"
          exit 1
        fi
    resource:
      cpu:
        cores: 1
        threads: true
      memory:
        size: 4GiB
    policy:
      timeout:
        execute: {{.BlastFetchTimeout}}
  run-blast:
    image:
      uri: {{.WorkflowContainer}}
    mounts:
      /data: data
      /scratch: scratch
    cwd: /scratch
    script: |
      #!/bin/bash
      {{.BlastCmd}} -num_threads {{.BlastCores}} -query "{{$run_name}}.fa" -db {{$dbpath}}/{{$dbname}} -out {{$run_name}}.blast.out {{.BlastOpts}} || exit 1
      cat {{$run_name}}.blast.out
    resource:
      cpu:
        cores: {{.BlastCores}}
        affinity: NUMA
      memory:
        size: {{.BlastMemory}}
    policy:
      timeout:
        execute: {{.BlastQueryTimeout}}
    depends-on:
      - name: fetch-db
        status: FINISHED

Adapt the parameters

  • Remove the following parameters: CustomBlastDbFetchCmd, CustomBlastDbOptions, CustomBlastDbName, CustomBlastDbTimeout, RetrieveQuerySequencesCmd, and BlastOutputPath
  • Add two new parameters:
    • Name: S3Uri Type: Text Description: AWS URI prefix for blast inputs and outputs. Default: A URI like s3://<bucket>/<path...>
    • Name: S3Secret Type: Text Description: Secret from the secret store with AWS credentials. Default: A URI like secret://<scope>/<name>

Run the workflow

Once you save the modified BLAST workflow template, you can run it the same way as the stock BLAST workflow template, supplying the new S3Uri and S3Secret parameters.