When My Imports Go One Case at a Time and Take Forever

Ruby 64-bit version should make this henceforth a rare circumstance, but with big data beating at our doors, the advice below remains useful.

——————————————————————————————————-

This can happen when there is not enough free RAM to hold, process and write more than a single case at a time.  It is much more likely to occur on a CPU and RAM intensive import, like blended SPS/ASC or SAV, than a non-normalised import format like SSS, Surveycraft or Dimensions.

You should never commence an import without a backup of \Casedata, job.ini and vartree.vtr.  If you have to end the Ruby process because an import will (at one case per pass) take hours to days, then the case data files will be in an inconsistent state, and will need to be restored.  Remember to clear \Casedata before restoring – otherwise new variables which are not overwritten can cause problems when you try the import again.

Solutions to this problem, in preferred order, are:

1. If the import format is SPS/ASC or SAV, consider using SSS instead. SSS is a fast and undemanding import format – as long as multi-response variables are discrete (and not dichotomous)
2. Close down all other running applications (Outlook, Excel, Skype, etc). You can do this while the import is running.  The freed RAM, if enough of it, will be used immediately by Ruby, and the cases per pass should jump to something sensible.
3. If that does not work, then reboot. This will give you a big slab of contiguous RAM
4. If a reboot does not help, then reboot again, then Control/Alt/Delete and close down unwanted processes, such as GoToMeeting, Apple I-Tunes Helper, etc – anything which is not OS sensitive (you should ask your local IT to provide a list of processes which can be safely ended within your standard setup – if in doubt, leave it alone!).

5. Try increasing the standard threshold from 70% to 75%.

The idea is to increase the gap between Write (75%) and Load (54%) as much as possible.  The only way to reduce the 54% is to close down other RAM hogs. 75% is danger territory on Windows 7, but you might be lucky depending on the “multi-response-as-dichotomous:total number of variables” ratio. Windows XP was safe at 85%, even 90%. That’s progress for you.

If that still does not work and the import format must be SPS/ASC or SAV, then contact us.  We have a dedicated SAV/SPS stand-alone importer which has handled up to 150,000 variables and 150,000 cases from a SAV file on a 4 gig machine.