Hello all.
Sorry if this is going to be long. I know having to debug other peoples scripts, I prefer a good concise explanation to figure things out! And I added a copy of some of the code that is giving me errors. It's just a lot of fluff you need not worry about.
I have been using IDL for a few years now, but my new job has forced me to start taking my more casual use of this language to a new degree. As such, I have found myself a bit baffled with how I can concatenate structures in IDL. Or more specifically, concatenate in a certain way to maintain the standards of a code that has been previously written (and perhaps without the proper foresight to prevent these issues.)
Essentially, I have two pre-existing data structures. (Included below, named complete_set and data_set) and I need to place data_set into complete_set. Normally, this isn't an issue. create_struct will concatenate these two just fine. However, I am now working with a file that is well over a gigabyte in size. data_set stores the needed information for the some 2 million indices. On older computers, some other users in my group can't directly concatenate this structure into the other, as the resultant strain on the memory is a bit too large.
I would delete other variables and try the use of temporary, but that's not really possible here. I'm not in the main function, and data_set is needed for other portions of the code before the step it dies at.
My idea was to essentially pop off an element of the data_set structure at a time, and place that into complete_set. This way when I redefine complete_set after each iteration, it takes less memory and I can then increase the memory being taken from swap for complete_set. Only I can't seem to do this. When using create_struct, I am stuck naming things (by virtue of not wanting to change a few year old huge suite of scripts):
complete_set = create_struct(complete_set,"data_set",data_set)
Replacing the full structure data_set with data_set[0] works fine. But when I try to run the command:
complete_set=create_struct(complete_set,"data_set",data_set[1])
I get issues because the sub structure data_set already exists. I would have thought it would concatenate an additional entry into that sub structure.
So my general question is, how might I start adding new elements to complete_set directly, one at a time.
Thanks for your help!
Brian.
~~~~~~~~~~~~~~~~~
IDL> print,complete_set
{ 1}
IDL> help,complete_set,/str
** Structure , 1 tags, length=2, data length=2, refs=2:
FLAG INT 1
I'm only copying the first element of data_set. Each entry is large, and all 2 million entries are essentially identical.
IDL> print,data_set[0]
{{ 0 0 0
}{
0
0 0 0
}{
0 0
0
}{
0 0 0
0 0 0 0 0 0 0 0 0 0
}{
0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0 0
0 0
}{
0
0
}{
0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0 0
0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0
}{
0
0 0 0 0 0 0
}{
0
0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0 0
0 0 0 0 0 0
}{
0 0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0
}{
0
0 0 0 0 0 0 0 0
}
0 0
}
IDL> help,data_set,/str
** Structure , 48 tags, length=459, data length=459, refs=3:
COLUMN1 STRUCT -> Array[1]
COLUMN2 STRUCT -> Array[1]
COLUMN3 STRUCT -> Array[1]
COLUMN4 STRUCT -> Array[1]
COLUMN5 STRUCT -> Array[1]
COLUMN6 STRUCT -> Array[1]
COLUMN7 STRUCT -> Array[1]
COLUMN8 STRUCT -> Array[1]
COLUMN9 STRUCT -> Array[1]
COLUMN10 STRUCT -> Array[1]
COLUMN11 STRUCT -> Array[1]
COLUMN12 STRUCT -> Array[1]
COLUMN13 STRUCT -> Array[1]
COLUMN14 STRUCT -> Array[1]
COLUMN15 STRUCT -> Array[1]
COLUMN16 STRUCT -> Array[1]
COLUMN17 STRUCT -> Array[1]
COLUMN18 STRUCT -> Array[1]
COLUMN19 STRUCT -> Array[1]
COLUMN20 STRUCT -> Array[1]
COLUMN21 STRUCT -> Array[1]
COLUMN22 STRUCT -> Array[1]
COLUMN23 STRUCT -> Array[1]
COLUMN24 STRUCT -> Array[1]
COLUMN25 STRUCT -> Array[1]
COLUMN26 STRUCT -> Array[1]
COLUMN27 STRUCT -> Array[1]
COLUMN28 STRUCT -> Array[1]
COLUMN29 STRUCT -> Array[1]
COLUMN30 STRUCT -> Array[1]
COLUMN31 STRUCT -> Array[1]
COLUMN32 STRUCT -> Array[1]
COLUMN33 STRUCT -> Array[1]
COLUMN34 STRUCT -> Array[1]
COLUMN35 STRUCT -> Array[1]
COLUMN36 STRUCT -> Array[1]
COLUMN37 STRUCT -> Array[1]
COLUMN38 STRUCT -> Array[1]
COLUMN39 STRUCT -> Array[1]
COLUMN40 STRUCT -> Array[1]
COLUMN41 STRUCT -> Array[1]
COLUMN42 STRUCT -> Array[1]
COLUMN43 STRUCT -> Array[1]
COLUMN44 STRUCT -> Array[1]
COLUMN45 STRUCT -> Array[1]
COLUMN46 STRUCT -> Array[1]
COLUMN47 STRUCT -> Array[1]
CRLF BYTE Array[2]
|