On Mon, Oct 2, 2017 at 4:37 AM, Jose Marques <[log in to unmask]> wrote:
>> On 29 Sep 2017, at 02:34, Nico Kadel-Garcia <[log in to unmask]> wrote:
>>
>> Storing individual messages in individual files, sacrosanct and unedited, is part of the basic IMAP specification.
>
> UW IMAP written by the author of the IMAP RFC stored multiple messages in the same file (one file per mailbox). It also offered the option of an indexed (binary) mailbox format for better concurrent access (less whole file locking). The way messages are stored on the server is an implementation detail. Having said that Exchange and Office 365 are awful in this regard.

I'm.... going to dig back into yesteryear for this stuff. Let me know
if it's uninteresting, or inappropriate here, and I'll try to scale
back such generalized reminescences.

That's not how I remember the client: the reference client for
wu-imapd was "pine", contained in the "wu-imapd" source code tarball
from Washington University. That had one directory owned by the user
for the entire local mailbox. That mailbox directory was defined as
$HOME/ in the original implementation of the server. Folders were
built as directories under the mailbox, and individual messages were
individual messages were individual files. The metadata about the
messages, such as whether they had been read, was encoded in the
filename, so that changes of message status were file renames and
extremely reliable atomic operations. This was a major performance
benefit to IMAP: having to open up bulky folders and manipulate data
about individual messages in them has always been a source of
corruption of many database backed mail systems. The pristine message
stores were invaluable for spam filter training when I did some work
bringing some spam software over to 64-bit.

The real performance problem was when people put too many messages in
one level of one folder. The kernel operation to list many thousands
of files in the same folder became increasingly tedious, but it was
really easy to split one folder into multiple folders to make it
manageable again. I remember having some long chats with people who
never, never deleted or organized their email and splitting up their
email by year into multiple folders for them, on the server side, just
by grepping the "Date:" lines.

The original publicly available wu-imapd software package, and the
pine code contained within it, did not include SSL or support for
IMAPS in its published releases. I got yelled at by Marc Crispin for
stealing the wu-imapd implementation code for IMAPS and publishing it,
which I *did not do*. I published pointers to places outside the US
where you could get SSL patches. That mattered because sharing your
IMAP passwords and content in clear text to a remote server was a
quite dangerous practice, and the wu-imapd published version had no
encryption. The hooks were there, the code was not. I was never sure
if it was Marc, or Washington University, who elected to avoid issues
with US export encryption regulations by excising the code, but it
created maintenance issues if you bolted on the third party SSL tools.
OpenSSL was safely available for download outside the USA, but it was
not being published by commercial operating system distributors at the
time, and I was working with SunOS.

I also kept publishing patches so that imapd and Pine, the client
included in the code base, could both be configured to use the same
$HOME/mail subfolder to store a particular user's messages. Marc was
absolutely convinced that a machine used for an IMAP service should
use $HOME/ as the base of the IMAP directory, because *of course* you
would never use a mail server as anything else or ever use an IMAP
capable email client on it. And there would never be anything in your
$HOME/ except your email, becuase of course you would never have
working user directories on such a host. Unfortunately, some of us
couldn't afford another server for just email. And as it was written,
if you told imapd to use $HOME/mail, Pine would save copies of that in
$HOME/mail/mail. The results were.... ugly, because using Pine on the
IMAP server itself would have IMAPD report that as a folder called
"mail/mail", which Pine would try to download locally as
$HOME/mail/mail/mail/, and hilarity would ensue.

My first college computer course was in Scheme, which had a navel
gazing fascination with recursion and self-recursion. This sort of
system destroying behavior is why I *loathe* careless recursion.

I published those patches for imapd and pine to gracefully share
$HOME/mail/ for about 10 years, and I'm very tempted to swear that
Marc kept deliberately rewriting the relevant 20 lines or so of code
in wu-imapd to break my patches with *every minor release*. As best I
could tell, the upstream patches to that code had no other effect. The
code stanzas were quite small and legible, the code was pretty good
that way. When the wu-imapd codebase switched to C++, I threw in the
towel, I didn't have the time to maintain my forks anymore and I no
longer had to maintain my own email servers by then.


> The University of St Andrews is a charity registered in Scotland, No. SC013532.