| 04 Apr 2012 10:44 PM |
|
|
Hi,
I'm new to IDL and have been hired as a developer to improve the efficiency of an existing IDL application that is used as part of an operational system.
I have pin pointed one majorly slow part of the program down to the contour procedure. I'm not sure if this extends to IDL as a whole but this procedure seems to make very poor use of multi core systems.
As an example the data we pass to the contour procedure is a two-dimensional array 4051 x 2032.
Now this is a large amount of data something like 8.2 million points, however for a modern system this should be nothing to worry about. The problem seems not to be with contour rendering the points but with the processing behind breaking this data into levels, I tested this by reducing the amount of level from 16 to 3 and the process finished significantly faster.
Now my point is these points are independent of each other and therefore the code that calculates which level a point fits into is perfect for parallel processing, and would make this procedure much faster on a modern system. Currently the procedure runs no faster on an intel core duo system than is does on a brand new Xeon 6 core system. If there is no planned improvements in the area of parallel processing optimisations for IDL then I have no option but to suggest the redevelopment of the application in python.
I'm happy to answer any question about my post, and would be interested to see what the IDL developers have to say on this matter.
Thanks,
Tim
|
|
|
|
Deleted User
New Member
Posts:  
| 05 Apr 2012 07:57 AM |
|
|
The contour procedure is VERY old. I think it was purchased from another company in the late 1970's or early 1980's, and even then it was a black box to IDL developers. The only one brave enough to open it up was David Stern, the founder of the company, and he did so only rarely. I'm pretty sure no one else has looked at it since David sold and left the company over 10 years ago. In any case, there is no chance that these old IDL graphics routines are going to be updated, and this is decidedly NOT a "modern system." The company is going in a different direction with yet another new graphics system, which they are having difficulty getting to work well enough to actually do science with. Still, this is clearly the future and there is no going back. If your bottleneck is this old contour procedure, then I think you really are stuck with what you can do in IDL.
|
|
|
|
Deleted User
New Member
Posts:
| 07 Apr 2012 06:45 AM |
|
|
Thanks for the replies,
I have already looked at Davids library and the contour procedure still uses the built in idl procedure at its core.
As for the contour IDL procedure not being updated, I dont think that is justifiable. The old graphics system has not been depreciated so there is no reason for not continuing to update it. The IDL graphics tutorial itself states "Each graphical system has distinct advantages in comparison to the others, and this chapter presents an introduction to the Direct and Object Graphics systems" it seems to me that both systems are being advertised with no push toward the new system.
Anyway its seems moving to Python is the smart choice, I don't see how continuing to pay a license fees for IDL licences can be justified. If exelis doesn't have the resources to updated these old procedures maybe they should consider open sourcing some of the code and let the community optimize the code for them rather than just sitting back and watching IDL slowly die.
|
|
|
|
Deleted User
New Member
Posts:
| 07 Apr 2012 09:05 PM |
|
|
Amen to that!
|
|
|
|
Deleted User
New Member
Posts:
| 10 Apr 2012 10:10 PM |
|
|
Just for the record. Using the contourf function of the open source python library Matplotlib I'm able to plot that same data in 2 seconds where IDL was taking upto 14 seconds.
Exelis really needs to get its act together.
|
|
|
|
Deleted User
New Member
Posts:21
| 13 Apr 2012 03:44 PM |
|
|
Hi Timothy,
I would be curious what your data looks like. Do you have a simple reproduce case, with some sample data?
In general, much of IDL is multi-threaded. This includes all of the math operators as well as many of the math functions like Total, FFT, Convol, etc. As a user, you have control over the number of CPU's that are used, the size of the arrays where threading is used, etc.
The graphics routines are not multi-threaded.
I don't know all of the details of your current IDL application, but I would be curious whether converting the entire application to Python would actually result in faster code. Perhaps the Contour procedure just appears to be a bottleneck because everything else is fast?
Cheers,
Chris
Exelis VIS
|
|
|
|
Deleted User
New Member
Posts:
| 15 Apr 2012 10:48 PM |
|
Hi Chris,
Thanks for the response. I'm happy to supply you with a .sav file with the test data I have been using. I have also made a piece of test code for you that replicates what our code is doing.
By the way the contour procedure doesnt just appear to be a bottle neck it is a bottle neck as you will see.
Let me know if you need any more details.
Thanks,
Tim
http://www.itsqueeze.com/.../04/contour_data.zip
http://www.itsqueeze.com/...contour_test_pro.zip
|
|
|
|
Deleted User
New Member
Posts:
| 16 Apr 2012 09:03 AM |
|
|
While I am sure there are faster contouring algorithms these days, there weren't many faster 30 years ago when the IDL contouring algorithm was written. This is a tough data set for it, with lots of points very close together. But, even just reducing the size of the data by a factor of 2 results in a contour plot that takes about 5 seconds (compared to about 22 on my machine for the complete data set) and is visually identical to the original contour plot. (There are very slight differences, if you treat the contour plots as images and subtract the two, but I can't distinguish the differences by eye.) Reducing the size by a factor of four gives a contour in 2 seconds, and although I can perceive slight differences now, they are not so noticeable as to affect the results in any way, I don't believe. Remember, we are visualizing data, not looking at the data itself. There is a difference!
That said, the bottleneck is not with the Contour command itself, but with the CELL_FILL algorithm, which you need to use if you are putting filled contours on maps to get the colors correct. If you change CELL_FILL to fill, the contour plot renders in under 5 seconds, even with the full data set. And, the filled plot looks VERY different from the plot using CELL_FILL. Since there is no map projection in this code, this confuses me! I could wave my hands a bit if a map projection was involved.
I'm curious how this is going to be explained. :-)
|
|
|
|
Deleted User
New Member
Posts:
| 16 Apr 2012 09:56 AM |
|
|
OK, here is what you do with this data set. You are going to display the data as an image, rather than as a filled contour plot.
Replace this line in your code:
contour,MOST_VALS,most_lons,most_lats, levels=levels,C_color=c_levels,c_labels = 0,/cell_fill
With these lines (I am using Coyote Graphics routines here, as I think you should be, too):
tvlct, 255, 255, 255, 239
levels = [0,levels]
scaledData = Byte(Value_Locate(levels, most_vals))
cgImage, scaledData + 239B, Position=[0.1, 0.1, 0.9, 0.8], $
XRange=[Min(most_lons), Max(most_lons)],$
YRange=[Min(most_lats), Max(most_lats)], /Axes
cgColorbar, /Discrete, NColors=16, Bottom=239, Ticknames=StrTrim(levels,2)
I really can't tell the difference between the two plots, and this one takes 0.17 seconds to render on my machine!
|
|
|
|
Deleted User
New Member
Posts:
| 16 Apr 2012 11:30 AM |
|
|
Here is another odd thing with this data set. If I use the FILL keyword instead of CELL_FILL, then the resulting contour plot doesn't look anything like the plot created with CELL_FILL. In fact, it looks like most of the colors are missing. But, if I simply make sure the most_vals variable is greater than or equal to zero (which it should be already!), then the plot looks *exactly* like the CELL_FILL plot and takes only about 4 seconds to plot!
most_vals = most_vals > 0.0
Very, very strange!
|
|
|
|
Deleted User
New Member
Posts:
| 16 Apr 2012 01:17 PM |
|
I wrote an article documenting all this weirdness today. I don't know how to explain the data "fix" that somehow got the FILL keyword working. There will be an automatic invite to the IDL Expert Programmer's Association annual meeting for anyone who can explain it. (This is a $250 value!)
http://www.idlcoyote.com/...cs_tips/fastfill.php
|
|
|
|
Deleted User
New Member
Posts:
| 16 Apr 2012 01:41 PM |
|
|
I'll take that challenge. The reason for the behavior is that there is are a substantial number of NaN values in this dataset. After restoring the save file, notice the following:
IDL> wreal = where(finite(data), countReal, COMPLEMENT=wnan, NCOMPLEMENT=countNan)
IDL> help, wreal, countReal, wnan, countNan
WREAL LONG = Array[6624759]
COUNTREAL LONG = 6624759
WNAN LONG = Array[1606873]
COUNTNAN LONG = 1606873
Next if you use the >= operator to ensure that all data are greater than or equal to Zero, which also brings any NaN values to Zero, you see that there are no longer any NaN values:
IDL> data >= 0
% Program caused arithmetic error: Floating illegal operand
IDL> wreal = where(finite(data), countReal, COMPLEMENT=wnan, NCOMPLEMENT=countNan)
IDL> help, wreal, countReal, wnan, countNan
WREAL LONG = Array[8231632]
COUNTREAL LONG = 8231632
WNAN LONG = -1
COUNTNAN LONG = 0
This may explain the difference in the two contour plots, before and after.
-Josh
Exelis VIS
|
|
|
|
Deleted User
New Member
Posts:
| 16 Apr 2012 03:15 PM |
|
|
Whoa! Exactly right! I even had a note on page 161 of my book explaining this "feature" of filled contour plots. (I might have to read that darn book!) Good catch!
|
|
|
|
Deleted User
New Member
Posts:21
| 16 Apr 2012 04:51 PM |
|
|
Hey David,
Great use of the VALUE_LOCATE function!
Just for fun, I converted the code to use "new" graphics:
restore, 'c:/contour_data.sav'
levels = [0,2,5,10,15,20,25,30,35,40,50,60,75,100,150,200,300]
nlevels = N_ELEMENTS(levels)
loadct, 12, /silent, RGB_TABLE=rgb
rgb[1, *] = [255,205,33]
rgb[2:15, *] = rgb[16*(INDGEN(14)+1),*]
start = SYSTIME(/SECONDS)
scaledData = Byte(Value_Locate(levels, most_vals))
im = IMAGE(scaledData, most_lons, most_lats, RGB_TABLE=rgb, $
XRANGE=[30,300], YRANGE=[-70,70], $
GRID_UNITS='degrees')
mc = MAPCONTINENTS(COLOR='white')
c = COLORBAR(RGB_TABLE=rgb[0:15,*], MAJOR=nlevels, $
TICKNAME=STRTRIM(levels,2), POSITION=[0.05,0.1,0.95,0.15])
elapsed_time = SYSTIME(/SECONDS) - start
print, "New graphics time: ", elapsed_time
This relies on the new colorbar functionality in IDL 8.2, so you'll have to comment out that line in IDL 8.1. The original direct graphics contour (with cell_fill) was taking 25 seconds on my machine. When I run the above code, it takes 2 seconds (that's after running it once to make sure all the code is compiled).
Cheers,
Chris
Exelis VIS
|
|
|
|
Deleted User
New Member
Posts:
| 16 Apr 2012 09:45 PM |
|
|
Hi David,
Thanks for that. Although I would prefur Exelis just improve the contour procedure this is an acceptable workaround. However I'm having one further problem you might be able to help me with. As I said I'm new to IDL so this may be a stupid question.
I've used your suggestion above and combined it with the map_image procedure as the contour is to be overlayed onto a map.
levels = [0,levels]
scaledData = Byte(Value_Locate(levels, most_vals))
img = MAP_IMAGE(scaledData + 239B, Startx,Starty,lonMin=Min(most_lons), $
latMin=Min(most_lats), $
lonMax=Max(most_lons), $
latMax=Max(most_lats), COMPRESS=1, /BILINEAR )
TV, img, Startx, Starty
The issue is now this new image just draws right over the map. Is there an easy way in IDL make a single colour (i.e black) transparent in an image so that the map can show through?
|
|
|
|
Deleted User
New Member
Posts:
| 17 Apr 2012 12:09 AM |
|
|
ok,I've found you transparent image function. However the map_image procedure won't take the result as input. Also if I do the transparent part after calling map_image I get a shadow like affect on the yellow part of the contour.
|
|
|
|
Deleted User
New Member
Posts:
| 17 Apr 2012 07:33 AM |
|
|
I don't think transparency is what you want. (Although I believe it's what you think you want.) Nor to I think you want to be fooling around with Map_Set and Map_Image, although the simple nature of those routines make them seductive. I think you want to set up your map projection in a professional manner using Map_Proj_Init and the other Map_Proj* routines. (Although, I have to admit, I have found lots of errors in even those routines, so I've written my own programs to work around these limitations.) Give me some more information (e.g., the map projection information you are using), and I'll show you how I would do it.
|
|
|
|
Deleted User
New Member
Posts:
| 17 Apr 2012 11:42 AM |
|
|
Hi Chris,
Well, I'm not sure about anything right now. I've tried to put the image I created on a map. I just decided to use a equirectangular map projection, but I couldn't get this to work at all with Map_Proj_Init. I've spent several hours on it. I'm completely confused. I finally got something to work with a Map_Set command and a cylindrical map projection. But I had to use a center longitude of -180 in the Map_Set call to make it work. NO idea why! You can see my efforts here:
http://www.idlcoyote.com/misc/map_test_cg.png
I really liked how your map projection fit the image. But it appears that half the map is missing! You can see what I mean in this picture, where I added a grid to the map. You can see both the grid and the continents on the right hand side of the date line are missing. (Compare with my image.)
http://www.idlcoyote.com/misc/map_test_fg.png
Since I generally think I know what I am doing with Map_Proj_Init I am extremely confused about why I can't make this work with that routine. I suppose it has something to do with that strange longitude I had to use in Map_Set to even come close. I've just always assumed you were using Map_Proj_Init in function graphics, but maybe that's not the case. Can you reveal what kind of map projection code you are using?
By the way, I was running this in the IDL 8.2 beta. Perhaps the entire map is shown in your latest version?
Cheers,
David
|
|
|
|
Deleted User
New Member
Posts:
| 17 Apr 2012 12:33 PM |
|
|
Chris,
On, hang on. When I cleaned up my code, it turns out I did know what I was doing!
Here is the code I'm using:
xrange =[x[0], x[4050]]
yrange = [y[0], y[N_Elements(y)-1]]
limit = [yrange[0], xrange[0], yrange[1], xrange[1]]
centerLon = (xrange[1] - xrange[0]) / 2.0 + xrange[0]
centerLat = (yrange[1] - yrange[0]) / 2.0 + yrange[0]
cgDisplay, WID=1
scaledData = Byte(Value_Locate(levels, data))
cgImage, scaledData, Position=[0.1, 0.1, 0.9, 0.8], /Erase
map = Obj_New('cgMap', 117, Center_Lon=centerlon, LIMIT=limit, /OnImage)
map->Draw
cgMap_Continents, /Fill, Color='Charcoal', Map=map
cgMap_Grid, Color='Charcoal', Map=map, /Label
cgColorbar, /Discrete, NColors=17, Ticknames=names
And here is the result:
http://www.idlcoyote.com/misc/map_test_map_init.png
David
|
|
|
|
Deleted User
New Member
Posts:
| 17 Apr 2012 06:53 PM |
|
|
Hi David,
Thanks for all your help. After some fiddling with drawing order and colours I now have it displaying as required. Your method of drawing the data has help improve the speed of the code in some parts from 36.139 seconds to 2.735 seconds that’s over 13x faster.
I must say though to the IDL dev's overall I'm not to impressed by IDL and the way in which the inbuilt library is left to stagnate. For this reason and others such as the poor GUI abilities such as no styled text box widget (i.e. the ability to change the colour, font, etc of text in a text box) I will still be recommending redevelopment to Python or some other language in future.
Thanks again,
Tim
|
|
|
|