Just used up 4TB external drive, looking for best storage solution

Hardware
1

In the past 4 years I have taken up the Astrophotography hobby. This has increasingly been more data hungry due to some of my multi pane mosaic projects. (One of which uses over 1TB of data). I already offloaded everything up to 2021 on a separate internal drive. The 1TB project from 2022 will not fit on the external backup drive. This puts my processing of the images on hold until I can find a storage solution.

I would prefer having all my images easily accessible on a “large” external drive as to make acquiring older projects easier if integrating them with a newer project.

Presently my data is on a tiny Crucial X6 4TB drive. This has worked great for portability. The registered Black Magic drive speed is considerably slower than what it seems should be possible (around 700mb/s) “but” I rarely notice a slowdown when working with my data either from accessing data or processing data on this external drive. (Using my 2018 MacBook Pro).

This brings me to one of the options that would somewhat future proof my hobby. Lacie makes a somewhat affordable large storage drive that uses Thunderbolt cables. LaCie 1big Dock 16TB . Since this is a mechanical drive the speed would be substantially less than a Solid State. (It is advertised at 280mb/s). I believe that if I did my data processing over the Lacie mechanical large storage drive that it would be abysmally slow, but likely all other tasks would not pose a problem. Perhaps I could store everything over the large drive but do my processing on my Crucial 6X SSD drive.

Another thought would be to figure out how Daisy Chaining works and perhaps purchase 1 powered 4TB stand in place SSD drive each year and have them chained to my MacBook so all of my data is accessible (perhaps 1 drive per each year of data). I am kind of ignorant in how this works so I don’t know whether purchasing something like a OWC drive enclosure would be necessary. In theory this would give greater speed and the affordability would be that you would need to only buy 1 drive per year.

What thoughts would you have towards future proofing the needs for a lot of storage data?

2

This depends how serious you are about this and also how much money you wish to spend.

You could have a working drive (SSD) and an archive drive (Spinning HDD) this would give you the speed when you need it.

You also have options (if you have the cash) to invest in a RAID array which allows you to have multiple drives, this makes it easier to upgrade your drives as you need to and compensates if one of the drives dies. Apple sells RAIDs made by PROMISE.

The next question for me though is how you’re backing this data up for when one of your drives fails?

3

If you want the most affordable high-speed access (versus storage/redundancy), you can get there with large spinning drives and using a RAID array.

But the RAID should be configured for disk striping (parallel use of all the drives for speed). This provides faster data access, but no data redundancy.

Then use a separate storage array configured for maximum capacity but lowest cost. That would be used for backup assuming you can tolerate the slower/batch process of occasionally backing up the modified data.

Also, and this depend greatly on your processing data flow and algorithms, the other way Enterprise computing achieves high data throughput is by using “in memory” databases which are loaded in and out of secondary storage, but processed entirely in ram.

Lastly, storage systems with very large cache memory as part of the design, again, depending on data flow, will dramatically speed-up data access speeds allowing very high density, but slower, rotating drives to be used.

4

Geoffaire: Good question; The truth of the matter is that I don’t presently have any backup for my portable external drive. Since this represents so much data this becomes a little problematic. Every day that I work with my data I know that I am taking a gamble. The fact is that I would be quite unhappy if I lost the data, but I am willing to lose this because it is just a hobby of which I can fill in the gaps of that which I have lost. Unlike losing actual family pictures would represent a real loss. (These are backed up on One Drive).

If I were to put some money out I could potentially back everything up with RAID. Another option might be to have a duplicate drive for each year for backup. Perhaps there are other low cost options?

At this point (unless easily solvable) I don’t want to lose my data but am willing to take the risk. Although this represents a real concern my bigger concern is to just get past the hurdle of being ground to a halt. I appreciate this part of the problem being brought up, if I can afford it I perhaps can take care of both problems at once.

5

I’d have a look at Backblaze, for a relatively cheap amount a month they will allow you to backup significant amounts of data. This gets it backed up somewhere other than your home.

Of course you’ll need a decent internet link.

6

SpivR: Would there be any specific drives that you are thinking of for a fast RAID drive and slower/cheaper large storage drives?

I have been looking at a few drives that I could afford and am wondering whether any of these would be a smart purchase.

I have already mentioned the Lacie 1big Dock 16TB . This would give enough storage for me for several years. Even though this is Thunderbolt cabled the speed is perhaps slow at 280 mb/s. Annoyances appear to be loud fan noise and the bright blue light.

There is an option that would be within my price range on the Apple accessory website. G-Technology 12 TB G RAID . In searching this product out, it appears that Raid 0 allows it to achieve speeds up to 500 mb/s. The negative for this product once again is the loud fan.

Sandisk makes an option that also fits within my price range, yet there seems to be some inconsistencies with issues on the Amazon review page. SanDisk Professional 12TB G-RAID 2 . For some reason it appears that the achievable speed of this drive is 400 mb/s which appears lower than the above mentioned G-Technology RAID drive.

There is a very inexpensive option made by Seagate. Seagate One Touch Hub 20TB External Hard Drive Desktop HDD. I have not stumbled on the data transfer speed but someone thought it likely was 100 mb/s.

I just noticed another product in the large storage and RAID category. I have never heard of Glyph, but they have been around for awhile. They have a reasonable priced drive that looks good to me but there is no Amazon reviews for the product. Glyph Production Technologies Blackbox PRO RAID Thunderbolt 3, 16TB It appears that their speed is quite good as they use the Thunderbolt cable and are primarily built for Macs.

I believe that any of the above might work out. Would anyone have any experience with the above drives or have any other specific recommendations?

7

Isn’t RAID 5 striped, and thus would provide (some additional) speed and redundancy?

Just be sure what you want to back up is direct attached to your Mac. While you can back up data from a Synology, for example, that does not come under the standard “Computer backup” plan, but a separate B2 plan. You can, however, back up drives that are not permanently plugged in, so long as you do plug in them in periodically, for long enough for Backblaze to scan and update the backup. I think it bugs me after about a month of not having a drive plugged in.

And this is another angle to consider. My approach to “mass storage” currently is a “drive toaster” that lets me plug in bare 3.5" or 2.5" hard drives or SSDs. Stuff I don’t want constantly lives on a motley collection of old drives repurposed from former external enclosures. Two of those drives are known to Backblaze.

8

I have purposely stayed away from RAID (Synology, et. al.) because I don’t want the heat, noise, and cognitive overhead of administering yet another system.

My storage needs so far are still at the point where I can juggle a few individual external drives without going crazy. That might change.

But I just started using 4 TB Crucial SSD USB-C drives - the tiny ones - and so far I am very impressed.

One thing you didn’t mention is how much data you must have online to use all at once. Is your data constantly growing year after year, or do you work with data in one-year chunks and then offline it?

The only advice on drive brands I can give is stay away from anything from Sandisk or WD. (WD owns Sandisk).

They are a very consumer-unfriendly company, conduct themselves unethically, and don’t service their customers.

Past scandal has been selling “shingled” technology hardrives as CMR drives and lying about it. (Shingled drives are cheaper and less reliable than CMR drives, but you can’t easily tell one from the other without advanced tools.)

Now many of their portable SSD drives, the Sandisk Extreme and Extreme Pro have been failing completely with inability to retrieve any data from the drive after very short time of use. These are the 2 TB and 4TB drives, but could be any of their models.

Sandisk has refused to acknowledge the problem, the replacement drives they sent out have also failed, and they have issued bogus firmware updates that don’t fix anything.

While this is going on, they started dumping inventory by selling current products on Amazon and other online stores for 75% off the regular discounted price without telling anyone about the problems.

Just a few weeks ago, a class action lawsuit was filed against WD/Sandisk.

I think enough warning, eh?

9

Wow, did someone buy them?

I have used and relied on WD hard drives and SanDisk memory cards for a lot of years now (over 20, I think). I don’t know about their customer service because not a single device has failed within its useful lifetime. This is in contrast to Seagate hard drives which basically covers all of the hard drives I have ever had fail. The aforementioned motley collection of drives comprise 10 WD units of varying sizes and ages and they have all been working fine. I even catalogued them all a couple of years ago which showed no issues. There are another 3 or 4 of the 2.5" “portable” drives in use among my and family members’ computers, too, though the newest is probably a couple of years old now.

1 Like
10

My Synology DS718+ is doing its job for 5 years now. I am using two Western Digital Red HDs inside of this thing. One of them is a WD40EFRX-68WT0N0, the other one is a WD40EFRX-68N32N0. The first drive mentioned predates the Synology. It is about 8 years old. The second one has replaced another WD40EFRX-68WT0N0. What is it that I have to do to administer the Synology? Nothing. It just works. I have set it up years ago, it is constantly updating itself. I never had an issue. Later on, I started using Homebridge in a Docker container. I am using Synology’s Video Station. It worked out of the box. Synology’s C2 Backup service is a very nice way to do cloud backups (I only do backup critical data to the cloud, about 900 GB). It just works either. Synology Drive apps are a great way to access my Synology from my iPhone and my iPads. They just work. And my Macs have zero issues mounting my shares using SMB. Those 4 TB in this RAID-1 are getting a bit crammed, I will have to replace those drives in the long term.

So, what about some hard drive data:

I very well might end up with another set of WD hard drives, given my good experience - and Backblaze’s stats.

And yes, I am another person using Sandisk Extreme SDs in my Canon camera - have been using them for years. Without issues.

I am sorry for anyone having a bad experience with Sandisk or Western Digital. And those examples you have mentioned really do sound bad. I am aware that I sound like a WD sales rep. But I honestly have not experienced any issues before and therefore I am quite a pleased WD customer…

:person_shrugging:

1 Like
11

Just a quick note on RAID levels and striping, since it came up earlier in this thread.

Yes, both RAID 0 and RAID 5 store data in stripes, but they do so quite differently.

A stripe is basically a chuck of disk space / data, of a fixed size (often based on the total drive size or the RAID implementation).

RAID 0: Think of a 2 drive RAID 0 array. Logically, the storage volume is laid out by alternating stripes between them two drives. Therefore, logically the volume looks like: Drive 0, Stripe 0 → Drive 1 Stripe 0 → Drive 0 Stripe 1 → Drive 1 Stripe 1, and so forth.

As a result, you can increase read and write speeds when accessing logical stripes in sequence, because the drive accesses are alternating between the two physical drives.

You can extend this to any number of physical drives.

Note that if ANY drive in the array fails, you lose the entire volume because there is no data redundancy.

RAID 5: Again, the physical drives are divided into stripes, and data is distributed across the physical drives. However, now, the corresponding stripes on each physical drive in the array are associated in this fashion: Let’s say you have three physical drives (the minimum for a RAID 5 array). For a given stripe, different pieces of data are stored in the stripes on 2 of the drives. On the third drive, “parity data”, which is basically a computation based on the data on the other 2 drives, is stored.

The effects of this are:

  • If any one drive fails, the array can be rebuilt once the drive is replaced, because the math involved in calculating the parity stripe is such that the data from any two of the three stripes can be used to rebuild the third. (This can be extended to any number of drives in the array, and you can actually have a split between the number of stripes containing data and those containing calculated “parity” data as well). Thus, RAID 5 provides protection against drive failure.
  • If you have N physical drives and use M of them for parity data, then your total storage available is (N-M) * physical drive capacity; so you are sacrificing some storage space for redundancy and data protection. Also, since stripes are matched up, all drives need to be the same capacity (some products, like Synology, provide software tricks to use drives of different sizes, but that is build on top of the basic RAID 5 design).
  • There is additional overhead in writing to a RAID 5 array because if you write a stripe on a physical drive, to recompute the new parity data you have to first retrieve the data in the same stripe on the other drives, compute the new parity, and then write the parity back out to the parity stripe. Therefore, RAID 5 generates a performance hit for writing. (There is a more subtle calculation in figuring out where data is located as well compared to RAID 0, where the computation is much simpler, but this aspect probably doesn’t reduce read performance all that much).

Hope this is useful, even though it has nothing to do with the ultimate solutions for the OP.

1 Like
12

To clarify my rambling last post:

TL;DR

SanDisk was purchased by WD.

That catastrophic data loss (and terrible customer service) has been with the palm-sized external USB-C SSD drives in the 2TB and 4TB capacity.

The fraudulent sales of SMR lower performance drives labelled as CMR (industry standard) drives has been the WD brand of 3.5 inch drives

The SanDisk memory cards, widely used by consumer and pro photographers, have not shown any issues or problems.

A large number of professional photographers, and consumers, are boycotting WD and Sandisk putting them on “never buy” lists. Not just because of the product failures, but the unethical, fraudulent, and poor customer policies.

There is, of course, some victim-blaming (not here), that lost data is because the user didn’t have proper backup, but I attribute that to the classic “you’re holding it wrong” type of emotional reaction to support a brand that has, in the past, been held in high regard.

With any problem, especially disk drives (since there are only a few big vendors), everyone will have a story of “it has always works great for me, never had a problem”.

Make your own informed choices. You can Google “WD sells SMR drives as CMR” and “class action lawsuit against WD” if you want to see more of the play by play.

13

[quote=“SpivR, post:8, topic:34814”]
One thing you didn’t mention is how much data you must have online to use all at once. Is your data constantly growing year after year, or do you work with data in one-year chunks and then offline it?
[/quote SpivR:

It would somewhat be a mix of the above but primarily I will pick a target and frame it in a way that would not be replicated if I shot the same target the next year. Likely I would only revisit old targets periodically, but I would certainly copy these over if they were not in my main working folder for that project.

So far in this thread I am getting the gist that the best route in choosing a hard drive solution would be in buying a large RAID spinning drives instead of any SSD solution. Additionally, there could be some concern about the disposition of Western Digital drives and as far as that goes perhaps Seagate Drives. Even though there have been people very pleased with both of these brands in conjunction with their Mac’s.

I personally can not rent any hard drive space as I am on a squeaky tight budget.

I can not afford anything too much over $800. With this in mind what specific hard drive would people here consider as the best choices? Would any the choices that I hi lighted above be a good option?

14

If your working set can stay within 4 TB, I would keep using external SSD like the Crucial and then get a very inexpensive large external drive for local backups.

If you need 8 TB working set, there are 8 TB SSD’s, but they are a bit pricey. However, the total ownership cost of buying a RAID chassis, dealing with docks or port expansion (if needed), and the admin/cognitive overhead of something not simply just another plug-in drive is worth considering in your budget decision.

In my day job, mid-range smart home/automation systems, we have a saying when clients want to stay at the lowest end of their budget “pay once, cry once”. Almost every time they go with the cheapest solution, is has to be upgraded sooner and the total cost when done (initial plus upgrade) ends up being more than just buying the more expensive, but more appropriate solution.

TL;DR if you can stretch to buy single SSD drives of the size you need, it will be a lot easier to work with, the fastest performance, and the simplest setup.

2 Likes
15

I finally see what you might be pointing out as far as getting 8tb Ssd as well as the RAID potential. OWC sells enclosures that can be filled with SSD storage and can be configured together with RAID.

The mini 4 bay would serve me well now but the next level up would remove some limitations.

Since there is about $250 in price difference I likely would go the cheaper of the two as I wouldn’t even be able to afford any SSD when I bought the more expensive model.

Since this has pass through capability I could later chain it with a larger HDD.

16

Just replying to double the number of astrophotgraphers in the thread! I posted this a while back: Astrophotography!

For the non-astrophotographers, this activity generates vast numbers of FITS files in the 30-50MB or more size range. FITS files are an astro-specific RAW format. They all have very long filenames full of metadata, and a perennial nuisance to deal with in column view. I never throw anything away because processing software is constantly improving and so going back to previous years and reprocessing images is often worth it.

File organization is key as it’s common to combine files from different years on the same target to make one image.

That’s for galaxies, star clusters and nebulae, collectively “deep space objects” or DSOs.

Planetary photography is very different. Gigabytes of video may be captured to create a single high resolution image.

Right now I’m managing with Samsung T5s and T7s. I’m in the process of making my MBP into a 6TB! The Samsung SSDs will go in adhesive pockets stuck to a clear case and connected by short cables. However, the short cables tend to cover up other ports like the Magsafe power socket and the HDMI socket.

Anyways, nice to meet you @AutumnSky!

1 Like
17

The NAS lifestyle is has a high entry cost, which is why I use my JBOD(IAP) model — Just A Bunch of Disks (In A Pile). It’s reasonably efficient to use my motley collection — 1 x 0.75TB, 3 x 1TB, 1 x 1.5TB, 1 x 2TB, 1 x 3TB = 10.25TB of storage for essentially zero cost, as they were all old drives previously replaced with larger and larger single drives.

To put 10TB in a RAID enclosure, I’d have to shell out something like the price of a new iPad Pro! Instead I spent a few dollars on a piece of software which works fine for me.

18

Hi Diane,

You are very correct that Astrophotography is a storage hungry hobby. That is quite impressive that you are in the process of making your MacBook into a 6TB storage device.

I think that I might know you from the Cloudy Nights forum (in specific the Astrophotography on a Mac Thread). On the Cloudy Nights forum I go by Paul in Northern Michigan. In this forum I am borrowing the name of my very energetic Golden Retriever. :slightly_smiling_face: Thank you for the Astrophotography link that you provided. This might prove to be helpful for more specific hobby related questions.

For those who might be following my storage quandary I have made my decision and have spent money on my 1st phase of a storage solution. I will be getting the OWC Thunderbolt 3 Mini enclosure. With this I will be filling it with 4 2TB 870 EVO drives. I am intending to use Raid 5 with these drives which would give a little redundancy in case of a drive failure.

Thank you all for your ideas and help!

2 Likes