One SSD had an internal short and turned into a space heater, luckily 
there was no fire. End excerpt.

Clearly, there is very poor safety engineering and/or quality control 
(as with certain Li batteries that did similar things in personal 
devices being operated by the user).  If that SSD had been inside a 
laptop (presumably, inside a rack mounted disk farm and there were fire 
extinguishers and possibly a machine room fire suppression system), 
things could have had a much worse outcome (most laptops have 
combustible materials).

As for the small amount of storage, the commentator is at a reasonably 
well funded (through government sources and possible tax-deductible or 
glamour philanthropy) HEP facility.  Much of the world, including 
non-collaboration funded university research facilities have rather poor 
funding at most entities within the USA (not all faculty members can be 
at Harvard, Stanford, etc.) -- administrative and some instructional 
facilities typically can get much more.  Many universities now outsource 
to paid "cloud" storage, with all of the issues that may entail.

On 8/10/21 3:08 PM, Konstantin Olchanski wrote:
> Hi, Larry, thank you for this information, it is always good to see
> how other people do things.
> 
> I am surprised at how little storage you have, only a handful of TBs.
> 
> Here, for each experiment data acquisition station, we now configure
> 2x1TB SSD for os, home dirs, apps, etc and 2x8-10-12TB HDD for recording
> experiment data. We use "sort by price" NAS CMR HDDs (WD red, etc).
> 
> All disks are doubled up as linux mdadm raid1 (mirror) or ZFS mirror. This is
> to prevent any disruption of data taking from single-disk failure.
> 
> (it is important to configure the boot loader on both SSDs to boot
> even if the other SSD is dead).
> 
> I am surprised you use 1TB HDDs. We switched to SSD up to 2TB size (WD blue SATA SSDs).
> 
> Failure rates of HDDs, the only reliable data is from backblaze:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.backblaze.com_b2_hard-2Ddrive-2Dtest-2Ddata.html&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A&m=NXYkiOfF7bPKBqi2iMgqsqrtLHRVdP7lIO-L5J4AmqQ&s=DgUuM1BVcm4jUkUWsi_DNMAjvkuy1zl1oaDQzrC4YAk&e=
> and
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.backblaze.com_blog_backblaze-2Ddrive-2Dstats-2Dfor-2Dq2-2D2021_&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A&m=NXYkiOfF7bPKBqi2iMgqsqrtLHRVdP7lIO-L5J4AmqQ&s=lPk2j2mTwp6uDzrZYUsP2rIxyRiacBHZOU0o7R5mUqM&e=
> 
> Failure rates of SSDs, seems to be very low, I only have 2-3 failed SSDs. One SSD had an
> internal short and turned into a space heater, luckily there was no fire.
> 
> For backups of os and home dirs we use amanda and rsync+zfs snapshots. Backups
> of experiment data is not our responibility (many experiments use usb hdds).
> 
> 
> K.O.
> 
> 
> On Tue, Aug 10, 2021 at 10:55:35AM -0400, Larry Linder wrote:
>> There are 25 systems in our shop, all linux based, a linux based server,
>> and synology Disk Station running raid 1.   The Disk Station has 12 TB
>> of space.  6 TB per for each raid level.
>>
>> We buy only one brand of disk with the black label.  They are typically
>> 1 TB.
>>
>> User boxes has a SSD drive for the OS and a 2 TB disk for the users
>> space and 32 G RAM. and a quad or six core AMD processor.  The graphics
>> boxes get a Video card with lots of ram.  3 D rendering on a slow video
>> care wast's a lot of users time.
>>
>> The server has a SSD for the OS and 6 TB for user apps /
>> library /usr/local and /opt.  It also has a mirror disk that keeps a
>> copy of the server locally.
>>
>> These systems are on 24 / 7 and accumulate a lot of hours.  No matter
>> what the make mechanical disks have a life span.  For grins I used to do
>> a post mortum on disk that failed.   There were to types of failures,
>> the spring that returns the arm holding the heads cracks.  The second
>> type of failure is the main bearings.  Newer disk seem to have less of a
>> bearing failure rate.
>> To prevent operational problems we just swap out the disk on each box at
>> about 5,000 to 7,000 hr.  The manufacturer says they are good for 10,000
>> hr. See the fine print in the Waranty,  You have to remember this is a
>> money making operation and down time is costly.
>>
>> Backups run at 12:29 and 0:29 in the AM.  At the end of the morning back
>> up a copy is sent to a remote site.
>>
>> For security we shut down the network at 6:20 PM, bring it up at 0:01 AM
>> and shut it down after back up is complete.  We bring it back up at 6:45
>> AM.
>> 10 yeas ago we had a fixed IP and the Chineese found it by just
>> continually pounding on the door.  The return IP was 4 hops to a city
>> north east of Shanghi.  They had installed a root kit on our server,
>> disabled cron.  When you changed the passwd to the server a few
>> millisecond later it was sent to china.  We got rid of the fixed IP and
>> reloaded all the systems.  So when you shout down the network to your
>> provider the next time your start it you get a different IP.
>>
>> We don't give the disks away as they contain a lot of design data,
>> SW,Cad programs, part programs for our mill etc.  We donate them to a
>> charity that drills the disks and recycles the rest.
>>
>> Larry Linder
>