X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 30 Jul 2015 02:42 PM by  anon
Best methods for spatially processing large images?
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:
New Member


--
30 Jul 2015 02:42 PM
    Note: I am using ENVI Classic and IDL I'm looking for some tips on the best way to complete some spatial processing on very large single band images. I'm investigating using ENVI's tiling routines to accomplish this, but have some issues. One example given in ENVI's help uses the tiling routines to process an image and save it in memory. It is something along the lines of: envi_file_query, fid, ns=ns, nl=nl, data_type=data_type pos=[0] mem_res = make_array(ns, nl, type=data_type) tile_id = envi_init_tile(fid, pos, num_tiles=num_tiles) for i = 0L, num_tiles-1 do begin data = envi_get_tile(tile_id, i, ys=ys) ; process the data here mem_res[0,ys] = data endfor envi_enter_data, data, .... This method is not working for me because the image that I am using is ~30k x 30k pixels. So when I try to make a blank array of this size using make_array(), it error out due to not enough memory. Is there a way to allocate more memory so that arrays this large can be created? This is the hangup I face with this method. The other method would be to write out the data to a file as it is being processed. Something like this: envi_file_query, fid, ns=ns, nl=nl, data_type=data_type pos=[0] openw, unit, outname, /get_lun tile_id = envi_init_tile(fid, pos, num_tiles=num_tiles) for i = 0L, num_tiles-1 do begin data = envi_get_tile(tile_id, i, ys=ys) ; process the data here writeu, unit, data endfor free_lun, unit envi_setup_head... This method would solve my memory issue. However, it is too slow for such a large image. To increase the processing speed I am thinking about using multiple IDL bridges to process multiple tiles at the same time. However, if I'm writing the file out, the parallel processing wouldn't work due to having to write out the data in the correct order. Any thoughts?

    MariM



    Veteran Member


    Posts:2396
    Veteran Member


    --
    03 Aug 2015 01:54 PM
    File read/write uses the system RAM mostly so one thing you can do is increase the tile size in the ENVI Classic preferences under the Miscellaneous tab. This will allow the tiling routines to use larger pieces of RAM to read/write the file and possibly increase the speed of the process. This would be best if your have significant RAM on the system (>4GB) and to set the tile size to no more than 5% of the physical RAM on the system.
    You are not authorized to post a reply.