X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 25 Apr 2013 02:22 PM by  anon
GLT creation timing depends on Customization settings?
 4 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:33
New Member


--
25 Apr 2013 02:22 PM
    Here's the deal: Several of us have noticed that, when running ENVI (4.8 or Classic) on 64-bit Linux systems in order to create GLTs, there is one item that helps to create GLTs many times faster. Strangely enough, it has to do with setting the "Customizations" and not using the defaults. See the help: Table of Contents -> Getting Started -> Getting Started with ENVI Classic ->Configuring and Customizing ENVI Classic -> Customizing ENVI Classic in aMultiple-User Windows (or Unix) Environment. The parameters that seem to matter are the various directory/folder paths; in particular the TEMP folder path. I'm not exactly sure why this happens; I do know that on 64-bit Linux systems, there is scratch/tmp file written to the TEMP directory when creating a GLT. What I don't understand is why changing the directory/folder to a different location leads to an ~ 5 (for the cases we looked at, keeping all other parameters the same) faster GLT creation time. And for the curious, the /tmp drive is a LOCAL drive, it is not on the network. This has been verified by us on 3 different 64-bit Linux computers, under both ENVI 4.8 and ENVI Classic, by three different individuals. We're happy for the increase in processing (especially when there are a lot of images) speed. But we've never read anything about this "feature" - benefit - of Customization in ENVI. Comments?

    MariM



    Veteran Member


    Posts:2396
    Veteran Member


    --
    29 Apr 2013 09:20 AM
    I don't know of any reason why ENVI would affect the speed of writing to the disk based on location. Are there a lot of files in the /tmp that could be cleaned up? Are there multiple users on the system writing to /tmp?

    Deleted User



    New Member


    Posts:33
    New Member


    --
    03 May 2013 09:11 AM
    Neither do we! We looked at the disks (same speed speed). In all cases, there were not a lot of files in /tmp, nor were the disks/parition that housed /tmp near full. And, no; in the cases we've found, there were not multiple users writing to /tmp. But, in all cases, choosing the default temp directory to be in users home directory made the GLT creation much faster.Multiple individuals have shown this to be the case.

    MariM



    Veteran Member


    Posts:2396
    Veteran Member


    --
    03 May 2013 10:26 AM
    Have you tried a time comparison by first going to a directory other than /temp and then within the same session running the GLT again writing to the /tmp? I am wondering if it is a 'first initialization' issue perhaps. Is this using the GUI or through a batch process?

    Deleted User



    New Member


    Posts:33
    New Member


    --
    07 May 2013 12:34 PM
    Certainly through batch (well, API, which strictly speaking isn't the same as batch). It is easy to put timing statements into programs when using the API. With the GUI, well, as you know, things are done just a bit differently, and so I can't put in precise timing statements; however, in this case it is easy enough since it takes so long. If you know how to do that when using the GUI, please share it with me! Also, is there an API in ENVI to get/set the various directories/folders (such as default tmp, default output, save_add), or is this only configurable via the envi.cfg file? Having said that, I have just tried from the GUI. And the GUI behaves the same way. 1) The output path is the same (modulo a slight name change). The temp folder is local in both cases. In the first case, it is /tmp. In the second case, it is ~/envi/temp, i.e., in the user's (my) home directory. The /tmp directory is on a partition that has 17 GB free (38% full) and the temporary file there is about 1.2 GB in size. No other files in the /tmp folder appear to be written while the GLT is being created, only the one associated with the GLT: envTueMay71250342013148_1.tmp-bsq. IDL is running at ~10%-30% of one of the many CPUs. Start time(date command), /tmp was: Tue May 7 12:50:35 EDT 2013 stop time (date command), /tmp was: Tue May 7 13:49:50 EDT 2013 2) Now, exit ENVI, change the default tmp to ~/envi/tmp; restart ENVI; verify that preferences reflect this change; open same IGM input file, set parameters identically, and see how long it takes; verified that temp file is present in ~/envi/tmp Start time (date): Tue May 7 14:00:59 EDT 2013 stop time (date): Tue May 7 14:10:39 EDT 2013 During the run with the default tmp set to ~/envi/tmp, the CPU% is much higher. It is like ENVI or the system is doing extra stuff when the default tmp is /tmp. 3) Now, continue by not exiting ENVI, but changing the default tmp back to /tmp, not saving configuration file. Verify that /tmp is being used by seeing the temporary file there (env*.tmp-bsq). Look at top, CPU % is ~11% again. Hmm. my prediction is that this is another hour. Start time (date): Tue May 7 14:17:18 EDT 2013 Stop time: I'm not waiting any more. It has been ~13 minutes and the process is somewhere between 1/4 and 1/3 done, at best. Therefore, it is not merely a "first initialization" issue. Factor faster when default tmp is ~/envi/tmp compared to /tmp: 9 min 50 sec vs. 59 min 15 sec: 6 times faster Really, you should be able to duplicate this at the Exelis Vis Galactic headquarters with all the information that I've given you. This is repeatable on our Linux computers (3 different computers administered by 3 different people), with three different people performing the tests, under both 4.8 and Classic. Presumably, it has something to do with how files to /tmp are written. While it may not be an ENVI issue (but maybe it is), it is something you should know about, and should recommend users to, for instance, follow the advice in "Customization". THIS TEST: Linux computer: Linux 2.6.32-358.6.1.el6.x86_64 #1 SMP Fri Mar 29 16:51:51 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux ENVI> print,envi_query_version() 5.0.2 ENVI> print,!VERSION { x86_64 linux unix linux 8.2.2 Jan 23 2013 64 64}
    You are not authorized to post a reply.