[ruffhi]

My prior FreeNAS build is still up and running ... very nicely, thankyou ... but I might have stuffed up the privileges somehow. I have a couple of stray folders and files that I can't edit / delete, etc.

Unfortunately, I can't just trash everything and start again ... as I would lose the 5-6TB of stuff that I have on the pool.

My solution ... a 2nd FreeNAS server that I can copy my files to, destroy the original and then copy my files back.

The other benefit of doing this is that my FreeNAS knowledge has expanded since I build my current server.

Here is my parts list ...

• CPU: Intel Core i3-6100 3.7 GHz Dual-Core Processor
• CPU Cooler: CRYORIG H5 Universal 65 CFM CPU Cooler
• Motherboard: Supermicro X11SSL-CF Micro ATX LGA1151 Motherboard
• Memory: 4 x Crucial 16 GB (1 x 16 GB) DDR4-2666 Memory
• Storage: 2 x Kingston A400 120 GB 2.5" Solid State Drive
• Storage: 3 x Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive
• Case: Corsair 550D ATX Mid Tower Case
• Power Supply: SeaSonic FOCUS Plus Gold 550 W 80+ Gold Certified Fully Modular ATX Power Supply
• Custom: 2 x CableCreation Internal HD Mini SAS (SFF-8643 Host) - 4x SATA (Target) Cable, SFF-8643
• Custom: 4 HDD Cage

The total comes to a touch over $1350.

More on these parts later.

 

[shilka]

The Fractal Design Ion+ 560 watt is tiny bit better and also cheaper than the Seasonic Focus Plus 550 watt on Newegg
Or did you already order everything?

 

[ruffhi]

The Fractal Design Ion+ 560 watt is tiny bit better and also cheaper than the Seasonic Focus Plus 550 watt on Newegg
Or did you already order everything?

Ordered, paid for and delivered. And now installed in the case. The only part I am waiting on is the dang nab it slow motherboard. The CPU had to come from California to Boston and it was here (basically) overnight. The motherboard is also coming form California ... but it is taking the scenic route.

 

[ruffhi]

First off, let me rip through the parts that are pretty much straight up and down ...

• CPU: Intel Core i3-6100 3.7 GHz Dual-Core Processor
• Memory: 4 x Crucial 16 GB (1 x 16 GB) DDR4-2666 Memory
• Storage: 2 x Kingston A400 120 GB 2.5" Solid State Drive
• Storage: 3 x Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive
• Power Supply: SeaSonic FOCUS Plus Gold 550 W 80+ Gold Certified Fully Modular ATX Power Supply
• Custom: Kingwin model KF-4001-BK 4 HDD Cage

The CPU is a plain i3, more than good enough for my testing server needs.  My main server has a Intel Xeon E3-1231 V3 3.4GHz Quad-Core Processor chip so this one will be in the testing rig.

The motherboard can take up to 64Gb ... so I maxed out the memory.  ECC is the critical element here.

The Kingston SSDs are my boot drives, two so that I can mirror them.

Three 4TB WD Red drives ... because a server without some storage is pretty dull.  I have a spare drive that goes into my main server when needed ... so the total pool for this testing server will be 4 x 4TB Red drives.  I will probably run them in RaidZ which means 3 drives for actual storage, 1 drive for redundancy.

Power supply already got a comment.  I just like Seasonic ... what can I say?  And Fully Modular.  The Fractal Design mentioned was PLATINUM efficient, which might have been nice.

Finally, the HDD cage.  I have two in my main server (Kingwin model KF-4001-BK).  I have removed the stock fan and replaced them with Gentle Typhoon 2150rpm 120mm fans.  That Kingwin model has been replaced by the Kingwin model KF-4002-BK that only has an 80mm fan ... but does provide options for power (either molex or sata).  I did originally get one of these via Amazon but ended up sending it back - mainly because of the on/off switches at the front.  I noticed that I was accidentally turning a drive off.  Not something that you actually want to do for real.

So ... I picked up a refurbished KF-4001-BK from newegg and a Gentle Typhoon 1450rpm fan.  We'll see if it is quieter.

 

[ruffhi]

• CPU Cooler: CRYORIG H5 Universal 65 CFM CPU Cooler

I am a little concerned that this CPU cooler is overkill. I would prefer the Cryorig M9i ... but the price of these have jumped from ~$30 to over $60. I asked about that here and someone mentioned the Trade War.

The H5 Universal is 853 grams with fan. The M9i is only 425 grams ... half the weight. No overclocking with the chip and not a lot to push it very far ... hmmm.

I also have an old Noctua that I could use. I can't even remember what type it is.

 

[ruffhi]

• Case: Corsair 550D ATX Mid Tower Case

This is my first computer case. I purchased it through IBuyPower many moons ago. The first time I saw it, I knew I wanted it. It just looked beautiful. It served as my workhorse until I upgraded (and went underwater) in a CaseLabs Mercury M5. It has more than enough to handle a 4 HDD server. And that is not even using the other 6 drive locations that I could pack in there.

 

[ruffhi]

• Motherboard: Supermicro X11SSL-CF Micro ATX LGA1151 Motherboard
• Custom: 2 x CableCreation Internal HD Mini SAS (SFF-8643 Host) - 4x SATA (Target) Cable, SFF-8643

Right ... time to discuss the motherboard. The FreeNAS forum hardware discussion document has this to say ...

X11SSL-cF
The Skylake equivalent of the X10SL7-F, the X11SSL-cF takes the X11SSL-F’s feature set and adds an LSI/Avago SAS 3008 SAS3 controller, for a total of 14 direct-attach drives. With its 64GB RAM limit, it is a natural choice for those servers that would be limited by the X10SL7-F’s 32GB RAM limit.

The X11SSL-CF uses the C232 chipset ... but has those SAS controllers available to add 8 additional drives. Hence the 2 x custom cables in my list of hardware.

This is the first time I have used SAS controllers and I am pretty sure that I have a lot of reading to do about these ... starting with

https://www.ixsystems.com/community/threads/crossflashing-a-x11ssl-cf.59048/
https://forums.servethehome.com/ind...-cf-o-board-i-ran-the-fw-update-before.19733/

 

[ruffhi]

My motherboard gets here tomorrow. Then I can stand up a temp pool on my new hardware. Copy my existing pool over and destroy my existing FreeNAS install and stand up a brand new one ... and copy my data back. This is / was going to take the 8 HDDs I have in my existing pool and the 4 HDDs in my temporary pool.

Except ... my existing pool reports a bad drive that needs replacing. Now my temp pool is down to three. I guess I could just stripe them and run the risk of a bad drive taking all of my data down ... no - I don't think I will do that. Maybe 3 HDDs in RaidZ will do the trick.

 

[ruffhi]

Motherboard (finally) arrived. I've plugged the power in, installed the CPU and the stock CPU cooler, added the memory ... where is the on-switch?

This is what happens when you get to use to user motherboards instead of server motherboards. Ok ... you are going to want the front panel cable ... are you? Where is the front panel connectors? And where is the useful (but very small) lettering to tell me which set of spikes is the power switch. Oh ... you are not even going to put that on the motherboard.

Sheeeze ... had to print a page from the manual (page 39, if you are interested).

 

[ruffhi]

Ok ... connected the power switch cable. I am able to turn the motherboard on. Worked straight off ... but boy-oh-boy does it take a while to boot up. I didn't have any bootable device connected so it ended up complaining about that. I pulled out my memtest USB, plugged it into the USB 3 connector in the middle of the motherboard (yes - this motherboard has a USB in the middle of it!) and now memtest is running on my 64Gb of ram.

I am still unsure how to get into the BIOS ... more reading of the manual to do.

 

[ruffhi]

Motherboard (finally) arrived. I've plugged the power in, installed the CPU and the stock CPU cooler

I just fired up IMPI access and it tells me my CPU is at 55° C. That is too hot by my book. I will install the Cryorig cooler and see how that does. I must say, the stock cooler is very quiet ... the fan runs at 1100rpm - which is pretty much nothing.

 

[ruffhi]

More Motherboard chatter ...

If you look at the picture above (bottom left of picture), you see 2 x yellow SATA ports and 4 x white SATA ports.
The yellow ones are special ... along with being data ports ... they also provide power

Why do you want power on a data sata port? So that you can use the SATA DOM (data on module) flash memory modules. I could imagine using two of these as FreeNAS boot drives. But I have my small SSDs already ... so maybe I will go this route in Rebuild #44.

Just to the right of the yellow sata ports are two stonking big black SAS ports. One port can read / write to 4 HDDs. My main server has 8 x 4TB HDDs in two 4-bay Kingwin cages. I could use those two SAS ports to connect up each cage.

Based on my googling, I have found that these ports come in two software modes ...

• IT = Target mode (HBA) - good for Storage Spaces Direct and ZFS - Software RAID
• IR - Raid Mode - Traditional storage redundancy - WBC/ HW RAID

That link above (two modes link) also contains this ...

Supermicro X11 or H11 onboard LSI 3008 are shipped with IR mode firmware ph.10. After few email exchange with Supermicro Support, here is the summary,

• its their official advisory that user can invalid their warranty if they want to flash the onboard LSI 3008 controller firmware to IT Mode.
• Supermicro also said the Chip does not support IT mode at all
• If user want to upgrade their onboard LSI firmware, it is best to RMA back to supermicro to do so.

This does not make sense at all, although my boards are not yet in production, but there is still needs to update the firmware to match the latest driver.

... and the link goes on to provide step by step instructions about how to flash them to IT mode.

There are some funny stories about people (trying to) flash to IT mode ...

• story about someone trying to flash from the wrong folder
  
• story about a fan being turned down because of the noise ... and the flow on impact
  
• story about running batch files without admin rights
  

The best part about this is that a) it gives you an insight into what people are struggling with and b) it gives you their solution.

The other useful think about other people's mishaps is that you can learn from them. The one about the wrong folder includes some screenshots that mention Avago Technologies. I just booted up this motherboard (RAM test into its second run through ... about 4hrs each) and the name Avago is all over the boot up process.

I was struggling to think where the guy was running these commands from ... maybe it is part of the Avago update process?

 

[ruffhi]

Just found this write up from my friends at the FreeNAS forum ...

https://www.ixsystems.com/community...shing-lsi-9211-9300-9311-hba-and-variants.54/

I am going to need a few hours, a good nights rest and some coffee to read through this.

 

[ruffhi]

So ... I have my motherboard (X11SSL-CF) sitting on its box, no drives added but with memory, cpu, cooler and power. The RAM is good ... 8 hrs of checking.

I installed it into the case ... only to find that the power connections are on the top ... and my 24-pin power cord is not long enough to go out the back ... in fact, my 550D doesn't have a hole up the top where the power cord can come back to the mother board.

I also installed the 3 HDDs that I have ... bootup - yay.

Then I entered SAS utility file ... only to find that my board and the LSI 3008 is already in IT mode! No flashing required. All that googling and reading for nothing.

 

[ruffhi]

FreeNAS installed. Users and Groups created. Pool created. Various DataSets created. Data copied over. Plex added in iocage jail, cron jobs set up, Acronis backup jobs updated. I think that is it.

I will cycle back and add some more comments to these steps. In a few weeks, I will take down both servers (old and new) and rebuild my go-forward version with the new motherboard and ram hardware.

 

[ruffhi]

Installing FreeNAS

There are lots of write ups about installing FreeNAS so I won't go into lots of details. A quick google revealed this one that uses rufus and a FreeNAS iso file. Follow it for the software install. The linked guide then goes into setting up FreeNAS but it uses the old interface. Read it, don't read it - up to you.

This is a short list of what I did.

• download rufus
• download iso file
• make bootable USB stick with iso
• plug USB into server PC
• boot from USB
• follow the bouncing ball to install

I installed FreeNAS to two 120Gb SSDs (mirror). The trick to mirroring your OS drives is to select both drives during the install process (ie target for FreeNAS install) and the software will chug along and mirror your install.

The one item that I would point out is that you should try to set a static IP address for your FreeNAS install. The software menu (not the IP interface) gives you an option to do that but I must be missing something because it always causes me problems. I ended up editing my router IP assignment feature and mapping my servers IP address based on its MAC address.

 

[ruffhi]

FreeNAS Users and Groups

I probably should reverse the items above to say 'Groups and Users' because I created my FreeNAS group first and the put the users in that group.

Log into your new FreeNAS server using the IP address given and the root password you created during the install. You can change the root password via the GUI by editing the root user and putting in a new password. Don't use the CLI (command line interface) as those changes are not saved and will be lost on reboot.

As you can see, I created a single group (FNUsers). I also created a few users ...

- FNRuff
- FNAcronis
- plex

The FNRuff user is my day to day user and will be the one I use for my share drive mapping. The FNAcronis user is for Acronis to access the server (back up purposes). The plex user is for plex.

 

[ruffhi]

I just got a cron job email about the SMART info for my drives. I set it up to be emailed at 5am ... it was emailed at 8am. Hmm ... I bet my system has the wrong time. Sigh ... it does. I change it to NYC time.

 

[ruffhi]

Pools and Datasets

FreeNAS is all about storage. And data redundancy. Given the importance on Pools and Datasets, I have found that you don't actually interact with them very much ... well, when I say interact, I guess I mean create.

This link has a very good intro to FreeNAS terminology and is well worth the read.

Initially, I put my 8 4TB HDDs into one pool using RaidZ3 (5 drives for data, 3 drives for data redundancy). There are a number of raid calculators out there ... this is the one that I used. With RaidZ3, my 32TB was reduced to 18TB. Given that I have extra drives sitting around in my 2nd server (testing), I decided that I can forgo the three drive redundancy and cut it down to 2.

With RaidZ2, my 32TB is now reduced to 22TB. Still - more than enough space for what I am currently using (about 5TB).

Pool Creation.
Storage, Pools, Add, Create new Pool ... I was going to show a pic here but I don't have any HDDs available for the new pool (they are all in the existing pool) so the screenshot is really dull.

Dataset Creation
I am still not sure about datasets. Do I create one big dataset and then create folders (in Windows) inside that dataset. Or do I create a handful of datasets and map different windows drives to each dataset. Or do I create one dataset ... and additional datasets inside that dataset.

My googling has revealed that there isn't a single approach that is suggested. Most people say 'do what feels good to you'. Great - very helpful.

Anyway ... here is what I did this time ...

• Created a single dataset for each Windows drive mapping that I wanted (with one exception)
• Datasets are ...
  • FreeNAS
  • General - Everything else that doesn't fit below
  • Manuals - I like to keep copies of the manuals for the stuff we have
  • Music
  • Pictures
  • PlexMedia - This is where my movies, TV shows, etc are that plex will use
  • PlexMetaData - This is a dataset that will house various plex meta data ... watched status of shows, any edits I make (eg sort order)
  • RecordedTV
  • Software - Storage for the software installs that I download
  • Temp

Another important aspect with datasets is permissions. I really screwed up the permissions in my first FreeNAS install. This was the primary reason that I decided to burn it all down and start again.

So ... 100% windows sharing will be used (see below) so ACL type is windows. Each dataset is owned by FNRuff and the FNUsers group ... with the exception of the PlexMetaData dataset which is owned by plex and the FNUsers group.

 

[ruffhi]

Sharing

I totally forgot sharing in my quick list of activities above. Again, plenty of guides about sharing. My experience is that you need to be consistent. I have a server, a couple of windows PCs (mainly W10, one W7) that need access. Wife uses a mac and I did set up a dataset for her to share on the old server ... but she never used it. As such, I will be 100% Windows SMB Shares now.

Each dataset got its own share. I decided to go with something very straight forward.