X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 04 Aug 2010 08:41 AM by  anon
gradual slowing down of process overnight
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:
New Member


--
04 Aug 2010 08:41 AM
    Hello, I set up a small IDL script that ran overnight while my machine was otherwise unloaded, and it smoothly slowed down in a systematic way as it ran. I am wondering what could be going on. Here's what the script is doing. I have a set of approximately 50,000 ascii data files gzipped on disk, all very nearly the same size. The IDL script reads in the list of these files, and then performs a for loop, which does the following: gunzip a data file to a temporary file read the data save the data in an IDL save file for faster access later delete the temporary file When the script started yesterday, it was processing about 61 files per minute. An hour later, it was processing 48 files per minute, and it kept falling off each hour until this morning, when I came in to work, and found it processing 21 files per minute. See the plot below. Given that the machine was otherwise unloaded in this time, I cannot think of a reason why this script would gradually slow down overnight. I run other programs that also process large amounts of data, so this question is not just driven by curiosity - I'd like to know if there's something going on that needs to be fixed. Thanks, Tom

    Deleted User



    New Member


    Posts:
    New Member


    --
    04 Aug 2010 11:48 AM
    I solved this. It has nothing to do with IDL. It has to do with disk access times in a directory that is filling up with a large number of small files. I will reorganize these data into a smaller set of large IDL save files instead of a larger set of small IDL save files.
    You are not authorized to post a reply.