Bug report: 503 Errors When Payload Exceeds ~37KB

[ShreyaKakhandiki]

Summary:
When using postHistory.js to submit dummy histories to the backend, I encountered repeated 503 Service Unavailable responses when the payload size exceeds approximately 37 KB. This appears to be a soft limit that is not explicitly documented in the API or Lambda configuration but consistently affects posting larger families.

Steps to Reproduce:

1. Find a FASTA alignment and dummy tree with many (>160 from my testing) sequences (enough to push payload size above 37 KB). PF0003 from the pfam database accomplishes this. The corresponding files are attached to this issue.

2. Run:

node aws/scripts/postHistory.js PF00003 PF00003.28_dummy.nh.txt PF00003.28.fa.txt shreyak

PF00003.28_dummy.nh.txt

PF00003.28.fa.txt

What I’ve Tried:

• Reduced the alignment to fewer (160 or less) sequences → POST succeeded.
• Confirmed that the timeout is set to 1 minute.
• Monitored CloudWatch logs: no signs of timeout or memory exhaustion in the Lambda logs for the failed case.
• Used identical command and tree/FASTA structure as successful runs — only the number of sequences (and resulting payload size) changed.

Hypothesis:

It may be that the evoldeeds-crud Lambda is either:

• Hitting a payload size or parsing threshold due to memory/CPU limitations (currently 128 MB).
• Failing silently before processing the full request, triggering a generic 503.

[ihh]

This is interesting. I wonder if the (extremely basic & un-optimized) JavaScript implementation of Felsenstein's algorithm is feeling the strain.

We could test some things about this locally using a command-line script (calcscore.js) which calls the same function (historyScore) as postHistory.js (the client script) and evoldeeds-crud.js (the Lambda script).

For example,

time node js/scripts/calcscore.js data/lg08evol.json PF00003.28_dummy.nh.txt PF00003.28.fa.txt

This takes 5.187s of user time on my 3.5GHz Mac Pro. The output ends in ,"score":-211320.97151423746}

Five seconds doesn't seem remotely long enough to trigger a Lambda timeout, but then 37k shouldn't trigger an API Gateway size limit either, hmmm

[ihh]

Confirm that I also get a 503 Service unavailable when trying to post the history using the web API.

Watching on CloudWatch while running on the command line using time, I see a few things:

• At 0:13, CloudWatch shows a message START RequestId: ... Version: $LATEST (the Lambda function starting)
• At 0:31, client reports a Status: 503 {"message":"Service Unavailable"} (presumably from API Gateway)
• At 1:15 CloudWatch shows the following messages:
  END RequestId:
REPORT RequestId: .... Duration: 60000.00 ms Billed Duration: 60000 ms Memory Size: 128 MB Max Memory Used: 128 MB Status: timeout

From this I think we can infer

• Log messages take about 13-15 seconds to get from Lambda to CloudWatch (at least START, END, and REPORT)
• The Lambda function (evoldeeds-crud.js) is hanging or timing out past 60s - maybe? But the API Gateway is timing out earlier? (Or, maybe the Lambda function is crashing in a way that the API Gateway interprets as a fail before Lambda calls it)

Fun...

[ShreyaKakhandiki]

POST /histories Lambda Timeout — Scoring Step Bottleneck

To debug the timeout issues when posting, I added internal logging to index.mjs to track:

• Size of the input payload
• Duration of each major step (parsing, validation, scoring)
• Memory usage at key checkpoints

Below is a table showing the results for three submissions of increasing size:

Case	Body Size	Validation Time	Scoring Time	Total Time	Max Mem Used	Outcome
50 sequences	9,240 B	81 ms	10.98 s	12.16 s	112 MB	Success
150 sequences	34,798 B	218 ms	26.38 s	30.44 s	125 MB	Success
Full file	152,227 B	603 ms	(Did not finish)	60.00 s	120 MB	Timeout

Example log lines:

INFO Scored in 26379 ms (cumulative)
INFO After scoring - Memory Usage (MB): { rss: '99.07', heapTotal: '42.05', heapUsed: '38.03', external: '2.49' }

Observations

• The scoring step is by far the most time-consuming and is the likely cause of timeouts.
• Parsing and validation complete quickly even for large payloads.
• Memory usage does not exceed the 128 MB limit, though it increases with input size.
• The full input never completes scoring within the 60-second timeout.

Suggestions

• Investigate whether the historyScore() function can be optimized or memoized.
• Consider offloading scoring to an asynchronous worker (e.g., SQS + ECS/Fargate).
• As a temporary fix, increasing Lambda timeout or memory might help for now.

[ihh]

Hmmm, if it is indeed computation time, then probably the best thing is to reimplement the scoring function in C++ or Rust

[ShreyaKakhandiki]

Hmmm, if it is indeed computation time, then probably the best thing is to reimplement the scoring function in C++ or Rust

Would you prefer C++ or Rust? If you let me know I can start trying to re-implement it and see if the speed improves.

[ihh]

I don’t think it matters too much; I’d probably use C++ because I know it better. If you’d like we can go through what the program should look like.

…

On Wed, Jul 30, 2025 at 4:36 PM ShreyaKakhandiki ***@***.***> wrote: *ShreyaKakhandiki* left a comment (evoldoers/evoldeeds#7) <#7 (comment)> Hmmm, if it is indeed computation time, then probably the best thing is to reimplement the scoring function in C++ or Rust Would you prefer C++ or Rust? If you let me know I can start trying to re-implement it and see if the speed improves. — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAANGYKFGVJC2YKEJN7VX4L3LFJI5AVCNFSM6AAAAACCVMPBYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMZYGEZDGOBQGA> . You are receiving this because you commented.Message ID: ***@***.***>

[ihh]

Essentially you'd be reimplementing calcscore.js but reading the history in as a CigarTree directly from a JSON file. The idea will be that the Lambda script will then pass data to this program (probably by saving the CigarTree and sequences as a temporary local file).

C++ is probably easiest. I'd probably use Nick Lohmann's JSON parser. I think you could get away without using a matrix library since the only matrix operations are multiplications and it's probably more efficient to just code row-column multiplication by hand.

[ihh]

It is probably worth just confirming that you can reproduce the timing behavior locally by running calcscore.js.

A quicker workaround might be to do some performance profiling of the JavaScript and use that to drive some data-guided optimizations. However in the longer term the C++ route will be faster.

The fiddliest algorithm involved is Felsenstein's pruning algorithm. I do have various C++ implementations of Felsenstein's algorithm kicking around (e.g. here) but I suspect they'll have too much baggage to save you real time; you might as well implement it yourself, it's a fun coding exercise.