Repository navigation
createRDBESEstObject() uses a lot of memory #233
Description
Activity
Whilst looking at this function I re-wrote the part of the code which handles sub-sampling - the function now runs faster (but still uses a lot of memory).
- added 6 commits that reference this issue
on May 13, 2025 I have now changed character columns to factors in the RDBESEstObject. I also added a new parameter called "incDesignVariables" - if this is set to FALSE then the design variables aren't included in the RDBESEstObject. This significantly reduces the size of the RDBESEstObject. Example:
myRawObject <- H1Example # Only use a subset of the test data myRawObject <- filterRDBESDataObject(myRawObject,c("DEstratumName"),c("DE_stratum1_H1","DE_stratum2_H1","DE_stratum3_H1")) myRawObject <- findAndKillOrphans(myRawObject, verbose = FALSE) myEstObject <- createRDBESEstObject(myRawObject,1, incDesignVariables = FALSE)Using the same H1 data as earlier the RDBESDataObject is still 917.2 MB. With the updated code, if I include the design variables the RDBESEstObject is 6,357.1 MB. If I don't include the design variables the RDBESEstObject is 3,359 MB.
- added a commit that references this issue
on May 14, 2025 @Kasia-MIR - do you want to see if these changes help when you are trying to create estimation objects from large amounts of data? They are in the dev branch.
@davidcurrie2001 really thank you for your changes. Fantactic job. The change in time is from 4 hours to 10 minutes... This is a huge change!
Reacted by David Currie
The function createRDBESEstObject() uses a lot of memory - this can be a problem when you are dealing with large sets of input data. For example, importing the H1 data for the NANSEA RCG creates an RDBESDataOject of size 971 MB. Running createRDBESEstObject() on that data then creates an RDBESEstObject of size 9630.6 MB - 10x the size of the RDBESDataObject.