Mail Troubleshooting Take 3

[seawasp]

I've posted screenshots of the output from "less" that I was able to cut-and-paste into a text file. The first shot is this one, called Less Screen One; this is typical of the first few screens of stuff that come up. Then the next few look like this; this unreadable garbage goes for several screens, then it goes back to things like Less Screen One for quite some time (I can't say for the rest of the file as I don't have the time to bounce through the entire 1+GB a screen at a time).

Also at intervals it pops up the error messages:

"less(3414) malloc: *** mmap(size=1073741824) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
less(3414) malloc: *** mmap(size=1073741824) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
:"

Now you see what I see... is it of any help?

Comments

[mouser.livejournal.com]

I can't tell if it's a relay causing it, or if the mail program is having the issue.



1) Unix/Linux message - End of line is a single character (CR) instead of the Windows stadnard (CR/LF). Probably something got confused in a configuration somewhere.

2) The second looks like part of a UUENCODED message (at a guess).

3) The memory allocation errors are because something doesn't understand the end of line character in #1 and is going until it runs out of memory at a one gig limit (one gig on a single line).

[mouser.livejournal.com]

^M = CR (Carriage Return)
^L = LF (Line feed)

That's what is getting munged.


I'm not sure what your system layout is - I only found one other post about your mail issues.

[seawasp]

Macbook, recent vintage.

[mouser.livejournal.com]

Unless you're using it as a mail server (which you don't seem to be), I'd contact your ISP. It seems to be having issues with handshaking with other mail servers.

[seawasp]

Note that the text you're looking at would be from 2004, so any issues handshaking would be like 5 years in the past.

[mouser.livejournal.com]

...I really need to read the entire thread...


Changing the ^M to your favorite line break while removing any other line breaks should reformat it correctly.

The stuff in the second view is definitely a UUEncoded message. These are a headache to "hand" decode with an external program.

I used to have a UUDECODE command line program but it won't run anymore under modern OSs (least of a Mac...)

[ninjarat.livejournal.com]

This is a job for MacPorts.
http://www.macports.org/

[mouser.livejournal.com]

...you were hearing theme music in your head when you posted that, weren't you?

[kpreid.livejournal.com]

Er, line feed is ^J.

[mouser.livejournal.com]

...maybe I just like a LOT of space between lines, did you ever think of that!?!?!





^L = FF (Form Feed/Page break)

[kpreid.livejournal.com]

Heh heh.^M^J
^M^J
I used to write software inside of a telecom app's scripting (y'know, modems, talking to BBSes and 'online services' kind of thing) and it got very familiar...^M^J

[ninjarat.livejournal.com]

Yes. Screen One confirms what I suspected: line ends are a disaster. See the line break between "MX-M" and "ozilla-Status:"? That shouldn't be there.

If that were my mailbox, what I would do is:
- Use m0rlock's perl script to convert the DOS end of lines to Unix.
- Break the file up into N-line pieces with split.
- Edit the first piece with Emacs and look at the line endings and pray that I can tell the difference between good line breaks (with white space or punctuation at the ends) and bad splits (no white space or punctuation).
- Create a replace expression or macro that deletes the line ends for bad splits and run that macro on every piece. I could do it in Perl or Python (or, sed for that matter) but for this kind of thing I want to see it work in real time.
- If I can't do it global then I figure out a way to repair broken headers as automatically as possible.
- Catenate all the pieces together again and load it up.

For the record, I've unmunged plenty of mail files before. This one takes the prize for weirdness for the mix of DOS and Unix end of lines and the misplaced ends.

Don't worry about Screen Two. That's MIME base64-encoded data, an attachment of some sort.

[seawasp]

Rgh. This will be... fun. Not. As I haven't used Emacs in 15 years either, and never even when I did did I use macros/expressions. And I don't do unix either (the only reason I've been able to try any of the things you've mentioned is that MacOS is, underneath, UNIX-based and so if I bring up Terminal I'm by default in Unix)

But the alternative is losing all that mail. Sigh.

[ninjarat.livejournal.com]

The only reason that I would use Emacs is because I'm familiar with doing this kind of work with Emacs. You can use whatever tools you are comfortable with. I could do it with Microsoft Word. In that case I would likely have to massage the existing line ends (paragraph marks in Word) to something unique like "lineendratPP", then change the bad ones, then change the good ones to something useful. The tricky part is figuring out how to either automate it or make it a few global replaces. Once you have that it's just tedious.

[seawasp]

The problem's going to be getting it into something that something will actually open. If Word would do it at all, I'd already have fixed it.

I tried what I think is "m0rlock's perl script" (perl -npe 's/\r//g' Inbox > NewInbox ), and after sitting for a while it came back with:

"Substitution loop at -e line 1, <> line 1."

[m0rlock.livejournal.com]

That's frustrating - I assume the file is either too big, or the line endings have it confused, or more likely a combination of the two.

Can you post a small chunk of the file somewhere? If you run

head -c 10000 Inbox > tmpInbox

it'll copy the first 10000 bytes of Inbox to tmpInbox. That way we'd have something we could experiment on. You can adjust 10000 down if you have privacy concerns, or up if you don't (a bigger sample would be better, of course).

[seawasp]

I've run that command. I'll email the tmpInbox to anyone who wants to take a crack at it (you'll obviously have to tell me an email addy to send it to); it's small enough that any mail program should handle it as an attachment with no problem.

Remember when working on it that this is a Netscape mail program running on an OSX Mac, in case that makes any difference.

[m0rlock.livejournal.com]

In the interest of defeating email address harvesting...send it to dmeyer@example.com. Except you should use dmeyer.net instead of example.com as the domain.

[kengr]

The garbage is standard mime base-64 encoding. So those are sections of the message that are images, or other "binary" content.

Odds are that the file is fine, it just needs to get run thru the right software to split out messages.