X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 04 Jan 2008 05:19 AM by  anon
reading more than 32767 characters in one line
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:
New Member


--
04 Jan 2008 05:19 AM
    For the analysis of log files, I have to read a file in which some lines have more than 32767 characters. Input line is too long for input buffer of 32767 characters. % READF: Error encountered reading from file. How can I acknowledge this error, skip this line, but continue processing the log file ?

    Deleted User



    New Member


    Posts:
    New Member


    --
    04 Jan 2008 05:19 AM
    To me I think it makes sense to CATCH this error, though you may have to do a fair amount of diagnosis after you CATCH it. The best way to handle it may depend on what follows this enormous "string". Thus, you can have a loop like this: openr, lun, inputFile, /GET_LUN nLines = file_lines(inputFile) fileText = strarr(nLines) currentLine = '' currentLineNo = 0 bytePosAfterLastRead = 0 errorCode = 0 for i = 0, nLines-1 do begin catch, errorCode if errorCode ne 0 then begin catch, /CANCEL if !error_state.code eq 'IDL_M_FILE_READERR' then begin print, 'READF error occurred at byte position ' + $ strtrim(bytePosAfterLastRead, 2) break endif else print, !error_state.msg endif else begin readf, lun, temp point_lun, -lun, bytePosAfterLastRead fileText[currentLineNo++] = temp endelse endfor free_lun, lunThe above loop will exit when it encounters the first READF error. I do not know how to keep the loop continuing because I do not think that IDL knows where the end of the extremely long text line is. Thus, when the loop exits in error you should have: a) the byte location of the first byte in the line that triggered the READF error b) the line number of the line that triggered the READF error c) all the text up to the line that triggered the READF error in an IDL string array When the loop exits you can proceed to read the rest of the file with READU using this approach: fileInfo = file_info(inputFile) nRemainingUnreadBytes = fileInfo.size - bytePosAfterLastRead remainingUnreadBytes = bytarr(nRemainingUnreadBytes) openr, lun, inputFile, /GET_LUN point_lun, lun, bytePosAfterLastRead readu, lun, remainingUnreadBytesAt this point the question remains: How do you parse the 'remainingUnreadBytes' byte array. If there are INDEED orderly text lines after the extremely long string, then you could search the byte array for the first newline character - STRING(10B) in IDL. That would help you find the byte location of the next orderly text line. You could then access the next orderly text lines either in your 'lun' file stream with a new POINT_LUN call, or in your byte array where you could parse out all your text strings by searching for newline chars and converting byte subarrays into strings with IDL's STRING function. James Jones
    You are not authorized to post a reply.