Mapper on large dataset

[jnukpezah]

Hi,
I am having issues running the giotto mapper implementation on a dataset of ~100,000 points with ~20 features. I used PCA as the filter function and the DBSCAN algorithm for clusterer and set n_jobs=-1. The dataset is scaled. I get a terminated worker error. Does giotto have constraints as to dataset size? Thanks
JN

[ulupo]

Thanks for the report. What sort of cover are you using?

[jnukpezah]

I am using a cubicalcover with n_intervals = 10 and overlap_frac=0.3.

[ulupo]

Thank you. This means that the output of the covering step is a 100'000 x 1'000 boolean array. This by itself does not seem like it should be a problem, memory-wise. It would be good to better understand at what point of the pipeline you get your system crash. One simple way would be to pass verbose=True to make_mapper_pipeline and see where you get the terminated worker error.

[jnukpezah]

Thanks Umberto. I will try that and see where that occurs and reach out again if needed. Thanks.