Hello,
I set up a small IDL script that ran overnight while my machine was otherwise unloaded, and it smoothly slowed down in a systematic way as it ran. I am wondering what could be going on.
Here's what the script is doing. I have a set of approximately 50,000 ascii data files gzipped on disk, all very nearly the same size. The IDL script reads in the list of these files, and then performs a for loop, which does the following:
gunzip a data file to a temporary file
read the data
save the data in an IDL save file for faster access later
delete the temporary file
When the script started yesterday, it was processing about 61 files per minute. An hour later, it was processing 48 files per minute, and it kept falling off each hour until this morning, when I came in to work, and found it processing 21 files per minute. See the plot below.
Given that the machine was otherwise unloaded in this time, I cannot think of a reason why this script would gradually slow down overnight. I run other programs that also process large amounts of data, so this question is not just driven by curiosity - I'd like to know if there's something going on that needs to be fixed.
Thanks,
Tom
|