Skip to content

Enable block gzip compression example #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sagehen03 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Enable block gzip compression example #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
14457d6
seems to work
sagehen03 Apr 18, 2023
32e78b8
removing unused import
sagehen03 Apr 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions bioindex/src/main/resources/compression.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/bash -xe

sudo aws s3 cp s3://dig-data-registry/hail.jar /usr/lib/spark/jars/
Copy link
Contributor Author

@sagehen03 sagehen03 Apr 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I built the hail.jar from the hail source (it needs to compiled using java 8 since that's what our EMR clusters use) and then uploaded it to S3. We probably need a better location in s3, but I used this one now since it's not in production use. This solution also relies on EMR continuing to put /usr/lib/spark/jars on the classpath.

Copy link
Contributor

@psmadbec psmadbec Apr 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for things like this we tend to use s3://dig-aggregator-data/bin/

1 change: 1 addition & 0 deletions bioindex/src/main/resources/geneExpression.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ def main():
# sort and write
df.orderBy(['gene']) \
.write \
.option("compression", "is.hail.io.compress.BGzipCodec") \
.mode('overwrite') \
.json(outdir)

Expand Down
5 changes: 2 additions & 3 deletions bioindex/src/main/scala/GeneExpression.scala → .../src/main/scala/GeneExpressionStage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
package org.broadinstitute.dig.aggregator.methods.bioindex

import org.broadinstitute.dig.aggregator.core._
import org.broadinstitute.dig.aws._
import org.broadinstitute.dig.aws.config.emr._
import org.broadinstitute.dig.aws.emr._

/** The final result of all aggregator methods is building the BioIndex. All
Expand All @@ -21,7 +19,8 @@ class GeneExpressionStage(implicit context: Context) extends Stage {

/** Use latest EMR release. */
override val cluster: ClusterDef = super.cluster.copy(
releaseLabel = ReleaseLabel.emrLatest
releaseLabel = ReleaseLabel.emrLatest,
bootstrapScripts = Seq(new BootstrapScript(resourceUri("compression.sh")))
)

/** Output to Job steps. */
Expand Down