Mail Issues #2...

[seawasp]

Well, I switched from Netrape Aggravator to the Mac Mail program for the interim.

So far, I can't get any of the obvious applications to open (a copy of) the original mail file. Word of course doesn't recognize it, TextEdit refuses it, and when I tried:

"formail -e -d -s inboxfile >> newinboxfile"

(substituting "Inbox" for "inboxfile" and "NewInbox" for "newinboxfile")

my console just sorta sat there for, well, a long time until I finally killed it off. There was a new file called "NewInbox", but it had nothing in it.

Any more ideas? I've got like 1.2GB of mail file sitting there mocking me. I'm sure MOST of it must be perfectly fine.

Comments

[laptop-mechanic.livejournal.com]

Have you tried thunderbird?

[seawasp]

I use Tbird for my News access.

[ninjarat.livejournal.com]

Fire up Terminal and edit the file with vi: "vi filename". The first few lines should look something like this:

From MAILER-DAEMON Wed Jul 29 18:45:59 2009
Date: 29 Jul 2009 18:45:59 -0400
From: Mail System Internal Data <mumble@mumble>
Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA
Message-ID: <mumble@mumble>
X-IMAP: 1172181894 0000005764
Status: RO

If they don't then either you have the wrong file or a compressed copy of the file. BTW, type ":q!" to quit without changing the file.

[seawasp]

Well (A) wouldn't it not necessarily start that way if it was corrupted?

and (B) I tried it and didn't get anything. I'm not sure if it worked at all; I haven't used Vile in about 15 years.

[ninjarat.livejournal.com]

Depends on the nature of the corruption. mbox files have a very specifically defined structure. Any deviation from that structure = your MUA (mail user interface -- the mail program) can't read it. That's why you need to go into the file itself and eyeball it to make sure you really have a valid mail file. If you didn't see anything with vi then just more the file. There should be *something* there.

[seawasp]

"more Inbox" gave me a screen of text interspersed with what looked like "^M" in inverse text. The screen of text looked like it was part of a message or messages.

[ninjarat.livejournal.com]

Okay, now we're getting somewhere. All those ^M (carriage return) characters are DOS end of lines. Need to get rid of them. Easiest way is with tr:

tr -d '\r' < Inbox > NewInbox
Do that then try to read NewInbox.  If that doesn't work then forcibly open the file with Text Edit and clear out any blank lines at the start of the file.  The first few lines of real message should look like what I posted before.

[seawasp]

Using that line above gets the following response:

tr: Illegal byte sequence

If I try Text Edit it refuses to open -- something about not being the right encoding.

[ninjarat.livejournal.com]

The heck? Okay, that's wack. It means you don't have a text file since tr only works on plain text. It doesn't like non-ASCII. I think I'm at the point where I'd have to look at the file to see what's wrong with it and how to fix it. Sorry. That file is really messed up somewhere.

Are you *absolutely* positive that you have the right file?

[seawasp]

Sure as I can be. Came from /Users/rykspoor/Library/Mozilla/Profiles/Sea Wasp/herpkqhe.slt/Mail/mail.sgeinc.com, and it's the only mail file of that size -- a size that makes sense as it's all my mail from 2003-2004 onward.

[m0rlock.livejournal.com]

The mostly equivalent command in perl to ninjarat's tr command is

perl -npe 's/\r//g' Inbox > NewInbox

It ought to be a little more effective on the non-text portions of your Inbox.

[ckd]

Hmm. Maybe not. There are way too many EOL styles.

CRLF - ^M^J - 0x1310 - this is the DOS/Windows standard as well as what most Internet protocols use "on the wire"

CR - ^M - 0x13 - The Mac standard from the days of Mac OS 9 and earlier; a fair number of programs still use this.

LF - ^J - 0x10 - The Unix standard since time immemorial or at least time_t 0. This is what formail is expecting.

What it sounds like is that the Netscape storage may have been using Mac EOLs (CR without LF).

seawasp: Did Mail's "File: Import Mailboxes" command not do the right thing?

[ckd]

(went back to original post: no, it didn't)

Hm.

I'd try replacing the ^Ms with ^Js, to see if that worked. Easy way:

perl -pe 's/\r/\n/g' < oldfile > newfile

[autopope.livejournal.com]

Did Mail's "File: Import Mailboxes" command not do the right thing?

Suggestion?

1. Download Thunderbird.

2. Tell Thunderbird to import the Netscape mailbox.

3. Then tell Mail.app to import the resultant TBird mailbox. (Mail.app is happy to import TBird mailboxes; I'm assuming TBird gave earlier Netscape users a slick upgrade path.)

4. Remove TBird.

[seawasp]

Well, I use Thunderbird for my News already.

Trying the "import", it gives me two options: Communicator 4.x and Eudora. I know Netrape 7 descends from Communicator, so I tried that; it gives no option to tell Thunderbird WHERE to find the files in question, but it went gray and seemed to be looking, so I left it alone. After about half an hour, it finally just went back to the two choices; examination did not show my mail in any folder.

[autopope.livejournal.com]

Pointing vi at a 1.2Gb mbox file is a really bad idea.

Instead, use "less" as in "less filename". Less is a pager, used to display one terminal screenful of a file at a time (with backward scrolling, regexp searching, and so on).

[seawasp]

Using "Less", it gives me a screen similar to the one I described with "more" (although I must emphasize that the "^M" characters in both the less and more conditions are in INVERSE VIDEO, so it's not just looking plaintext).

However, once I get that screen, "less" stops, no keystrokes seem to work until I hit a control-C, and then I ended up getting:

"less(3228) malloc: *** mmap(size=1073741824) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
less(3228) malloc: *** mmap(size=1073741824) failed (error code=12)
"
several times, sometimes with a single line of the earlier garbage in between.

I couldn't get out of "less" and finally killed the Terminal window.

[m0rlock.livejournal.com]

Your formail command is broken. formail doesn't take an input file as an argument - it operates on stdin. So what you probably want is something more like

formail -e -d -s < inboxfile >> newinboxfile

Which probably won't help, but at least it'll do something.

[seawasp]

It did the same thing as the "broken" command; sat there for a long time until I finally killed the window, and made a file called newinboxfile, but there was nothing in it.

[kpreid.livejournal.com]

Suggestion: Take the top of the file (say as displayed in 'less', which will replace the ^Ms etc.), drop it in a text editor. blank out any personal information, and show it to us, who might then be able to recognize the format or the way it's mangled.

[ninjarat.livejournal.com]

Yeah. I think we're at the point where we're not going to figure it out until we see the file. Something is broken and the guessing game isn't getting it unbroken.

[seawasp]

If I could actually **SEE** the file (i.e., get it into a text editor) I'm pretty sure I could fix it -- or verify that it was completely screwed.

Alas, none of my text editors want to read it. Which seems utterly bizarre to me; I mean, even if it was a completely random string of bits, if I tell a plain-text editor to look at it, shouldn't those random bits unparse, as ASCII, into a jumble of text?

[ninjarat.livejournal.com]

Not if it isn't ASCII. That's the rub. Most text editors can't deal with non-text data, and how they deal with it varies. Anyway, there's always Emacs. It will read just about anything. Size might be a problem but content isn't. For a quick check with Emacs, "cat Inbox | head -1000 > tempfile" and then "emacs tempfile". (And for the record, I hate Lj's auto-formatting. It eats > and <.)

[seawasp]

Hmm. I had thought that if I just took 8-bit strings of 1 and 0, I'd *HAVE* ASCII, even if it wasn't MEANT to be ascii.

Typed in that line; after a while, it came back to the prompt, I typed "emacs tempfile", and it comes up "maximum buffer size exceeded."

[ninjarat.livejournal.com]

ASCII is 7-bit, codes 0-127.

That error confirms something: line ends are messed up. So do this instead: "dd if=Inbox of=tempfile bs=1024 count=1024". That will give you an exactly 1MB tempfile. Edit that with Emacs. That will work even if the file is totally mangled. It may not be pretty but it will work.

[kpreid.livejournal.com]

You misunderstand:
Open the file with 'less' as you did before. Make your terminal window reasonably large. Select and copy from your terminal and show us the result.

[seawasp]

No, I don't misunderstand; I made a post showing that stuff.