Skip to content

Update firehose client to accept start param and remove keep-alive#1

Open
Harsh24893 wants to merge 1 commit intoreddit:mainfrom
Harsh24893:main
Open

Update firehose client to accept start param and remove keep-alive#1
Harsh24893 wants to merge 1 commit intoreddit:mainfrom
Harsh24893:main

Conversation

@Harsh24893
Copy link
Copy Markdown

💸 TL;DR

  • Adding start option to the client. Start can take in a timestamp or id of the last event received. If it is empty, it ll start streaming from the current timestamp
  • Removed keep alive as it is available by default

🧪 Testing Steps / Validation

Tested by npm start and fetching in a token id and subscription id

Copy link
Copy Markdown

@adamthesax adamthesax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few changes. Also can you verify we still need this:

// keeps node process alive to restart the stream	// keeps node process alive to restart the stream
setInterval(() => {}, 1 << 30);	setInterval(() => {}, 1 << 30);

Not entirely sure if it's needed

Comment thread src/index.js

program.option('--keep-alive');

program.parse();
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Let's lean into command line arguments and away form environment variables

It makes it easier to understand what the program needs (--help) versus reading docs/docs

Comment thread src/index.js
},
params: {
keep_alive: program.opts().keepAlive,
start: process.env.START,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: This will always reconnect to whatever start value is passed in leading to duplicate messages. What we want to do is keep start up-to-date with the last Id seen.

Something like:

let start = program.opts.start

async function listen() {
  const stream = await initStream();
  stream
    .on('data', (chunk) => {
        const data = JSON.parse(chunk.toString());
        start = data.id
        })
 }

async function initStream() {
...blahblahblah...
   params: { start }
}[

@adamthesax
Copy link
Copy Markdown

One other thought is we should move away from assuming a chunk is a message and use a line reader for this. See readline

It's more accurate/robust and better represents our protocol.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants