feat: Allow pullHourly for adapters that support granular data fetching (version 2) by 0xpeluche · Pull Request #11130 · DefiLlama/defillama-server

0xpeluche · 2025-12-19T09:28:25Z

The following PR requires the changes introduced by the PR: DefiLlama/dimension-adapters#5027
This is currently still a WIP, with a codebase that can run locally in order to adjust and validate tests

Required

List of environment variables required to run tests locally (this creates the slices locally as well as the daily data from those slices):

$env:DIM_PROTOCOL_NAME="aave-v3"
$env:DIM_ADAPTER_TYPE="fees"
$env:DIM_DRY_RUN="false"
$env:DIM_LOCAL_STORE="true"
$env:DIM_LOCAL_STORE_DIR="./dim-local-store"
$env:runLocal="true"
$env:DIM_SKIP_HOURLY_CACHE="false"
$env:DIM_SKIP_REFILL_YESTERDAY="true"

To do this, the desired adapter must be adjusted by adding the flag: pullHourly: true

Command to execute: npx ts-node storeAll.ts

Brief overview

The changes introduced by this PR aim to both speed up the execution of our internal jobs and improve the reliability of the returned data by providing higher granularity and better accuracy

Adapter updates are split into hourly slices (one slice per hour)
When a job runs, it pulls from the database the 23 slices preceding the current slice being processed
If all 23 previous slices are available, the job fetches the 24th (current) slice on-chain, and the rolling 24h daily data can be reconstructed from these slices
If any of the previous 23 slices are missing, the job fetches the missing on-chain data together with the current slice. Once all 24 slices are available, the rolling 24h daily data is rebuilt

g1nt0ki · 2026-01-09T18:33:54Z

defi/src/adaptors/db-utils/db2.ts


 let isInitialized: any

+const isLocalStoreEnabled = () => process.env.DIM_LOCAL_STORE === 'true'


this entire block is related to mocking the db function right, I would keep it out of this file, you can create a localDb file and keep all the logic there and add the logic for switching based of env flag where it is imported, tbh, I wouldnt this, we can start using a shadow/dev db for these testings, it is easier

g1nt0ki · 2026-01-09T18:45:10Z

defi/src/adaptors/db-utils/db2.ts

+  return `${yyyy}-${mm}-${dd}-${hh}`
+}
+
+export async function getHourlySlicesForProtocol({ adapterType, id, fromTimestamp, toTimestamp, transform = a => a }: { adapterType: AdapterType, id: string, fromTimestamp: number, toTimestamp: number, transform?: (a: any) => any}) {


can you add a bulk fetch method, that will pull all the hourly slices of all the protocols in the past 50 hours (50 because the hourly runner will try to refill yesterday's data if missing in the db)

then on the actual runner if requested time range is within this cache, we return from cache, else query the db

g1nt0ki · 2026-01-09T18:49:01Z

defi/src/adaptors/handlers/storeAdaptorData/storeAll.ts

 console.log('This will run with MAX_RUNTIME:', MAX_RUNTIME / 60000, 'minutes');
+if (isDryRun) {
+  console.log('>>> DIMENSIONS DRY-RUN MODE ENABLED <<<');
+  if (singleAdapterTypeEnv) console.log('Limiting to adapterType:', singleAdapterTypeEnv);


think we already have support for dry run, same for passing certain adapters to run, check ui-tool/dimensions.ts

for testing, I would say, create a test.ts script in this folder instead of editting this file

g1nt0ki · 2026-01-09T18:55:21Z