ZFS Cheatsheet


This is a quick and dirty cheatsheet on Sun’s ZFS

Directories and Files
error messages /var/adm/messages
DEGRADED One or more top-level devices is in the degraded state because they have become offline. Sufficient replicas exist to keep functioning
FAULTED One or more top-level devices is in the faulted state because they have become offline. Insufficient replicas exist to keep functioning
OFFLINE The device was explicity taken offline by the “zpool offline” command
ONLINE The device is online and functioning
REMOVED The device was physically removed while the system was running
UNAVAIL The device could not be opened
Scrubbing and Resilvering
Scrubbing Examines all data to discover hardware faults or disk failures, only one scrub may be running at one time, you can manually scrub.
Resilvering is the same concept as rebuilding or resyncing data on to new disks into an array, the smart thing resilvering does is it does not rebuild the whole disk only the data that is required (the data blocks not the free blocks) thus reducing the time to resync a disk. Resilvering is automatic when you replace disks, etc. If a scrub is already running it is suspended until the resilvering has finished and then the scrubbing will continue.
ZFS Devices
Disk A physical disk drive
File The absolute path of pre-allocated files/images
Mirror Standard raid-1 mirror
Raidz1/2/3 ## non-standard distributed parity-based software raid levels, one common problem called “write-hole” is elimiated because raidz in ## zfs the data and stripe are written simultanously, basically is a power failure occurs in the middle of a write then you have the ## data plus the parity or you dont, also ZFS supports self-healing if it cannot read a bad block it will reconstruct it using the
## parity, and repair or indicate that this block should not be used.

## You should keep the raidz array at a low power of two plus partity
raidz1 – 3, 5, 9 disks
raidz2 – 4, 6, 8, 10, 18 disks
raidz3 – 5, 7, 11, 19 disks

## the more parity bits the longer it takes to resilver an array, standard mirroring does not have the problem of creating the parity
## so is quicker in resilvering

## raidz is more like raid3 than raid5 but does use parity to protect from disk failures
raidz/raidz1 – minimum of 3 devices (one parity disk), you can suffer a one disk loss
raidz2         – minimum of 4 devices (two parity disks), you can suffer a two disk loss
raidz3         – minimum of 5 devices (three parity disks) , you can suffer a three disk loss

spare hard drives marked as “hot spare” for ZFS raid, by default hot spares are not used in a disk failure you must turn on the “autoreplace” feature.
cache Linux caching mechanism use what is known as least recently used (LRU) algorithms, basically first in first out (FIFO) blocks are moved in and out of cache. Where ZFS cache is different it caches both least recently used block (LRU) requests and least frequent used (LFU) block requests, the cache device uses level 2 adaptive read cache (L2ARC).
log There are two terminologies here

  • ZFS intent log (ZIL) – a logging mechanism where all the data to be written is stored, then later flushed as a transactional write, this is similar to a journal filesystem (ext3 or ext4).
  • Seperate intent log (SLOG) – a seperate logging devive that caches the synchronous parts of the ZIL before flushing them to the slower disk, it does not cache asynchronous data (asynchronous data is flushed directly to the disk). If the SLOG exists the ZIL will be moved to it rather than residing on platter disk, everything in the SLOG will always be in system memory. Basically the SLOG is the device and the ZIL is data on the device.
Storage Pools
displaying zpool list
zpool list -o name,size,altroot

# zdb can view the inner workings of ZFS (zdb has a number of options)
zdb <option> <pool>

Note: there are a number of properties that you can select, the default is: name, size, used, available, capacity, health, altroot

status zpool status

## Show only errored pools with more verbosity
zpool status -xv

statistics zpool iostat -v 5 5

Note: use this command like you would iostat

history zpool history -il

Note: once a pool has been removed the history is gone

creating ## You cannot shrink a pool only grow it

## performing a dry run but don’t actual perform the creation (notice the -n)
zpool create -n data01 c1t0d0s0

# you can persume that I created two files called /zfs1/disk01 and /zfs1/disk02 using mkfile
zpool create data01 /zfs1/disk01 /zfs1/disk02

# using a standard disk slice
zpool create data01 c1t0d0s0

## using a different mountpoint than the default /<pool name>
zpool create -m /zfspool data01 c1t0d0s0

# mirror and hot spare disks examples, hot spares are not used by default turn on the “autoreplace” feature for each pool
zpool create data01 mirror c1t0d0 c2t0d0 mirror c1t0d1 c2t0d1
zpool create data01 mirror c1t0d0 c2t0d0 spare c3t0d0

## setting up a log device and mirroring it
zpool create data01 mirror c1t0d0 c2t0d0 log mirror c3t0d0 c4t0d0

## setting up a cache device
zpool create data 01 mirror c1t0d0 c2t0d0 cache c3t0d0 c3t1d0

## you can also create raid pools (raidz/raidz1 – mirror, raidz2 – single parity, raidz3 double partity)
zpool create data01 raidz2 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0

destroying zpool destroy /zfs1/data01

## in the event of a disaster you can re-import a destroyed pool
zpool import -f -D -d /zfs1 data031

adding zpool add data01 c2t0d0

Note: make sure that you get this right as zpool only supports the removal of hot spares and cache disks, for mirrors see attach and detach below

Resizing ## When replacing a disk with a larger one you must enable the “autoexpand” feature to allow you to use the extended space, you must do this before replacing the first disk
removing zpool remove data01 c2t0d0

Note: zpool only supports the removal of hot spares and cache disks, for mirrors see attach and detach below

clearing faults zpool clear data01

## Clearing a specific disk fault
zpool clear data01 c2t0d0

attaching (mirror) ## c2t0d0 is an existing disk that is not mirrored, by attaching c3t0d0 both disks will become a mirror pair
zpool attach data01 c2t0d0 c3t0d0
detaching (mirror) zpool detach data01 c2t0d0

Note: see above notes is attaching

onlining zpool online data01 c2t0d0
offlining zpool offline data01 c2t0d0

## Temporary offlining (will revent back after a reboot)
zpool offline data01 -t c2t0d0

Replacing ## replacing like for like
zpool replace data03 c2t0d0

## replacing with another disk
zpool replace data03 c2t0d0 c3t0d0

scrubbing zpool scrub data01

## stop a scrubbing in progress, check the scrub line using “zpool status data01” to see any errors
zpool scrub -s data01

Note; see top of table for more information about resilvering and scrubbing

exporting zpool export data01

## you can list exported pools using the import command
zpool import

importing ## when using standard disk devices i.e c2t0d0
zpool import data01

## if using files in say the /zfs filesystem
zpool import -d /zfs

## importing a destroyed pool
zpool import -f -D -d /zfs1 data03

getting parameters zpool get all data01

Note: the source column denotes if the value has been change from it default value, a dash in this column means it is a read-only value

setting parameters zpool set autoreplace=on data01

Note: use the command “zpool get all <pool>” to obtain list of current setting

upgrade ## List upgrade paths
zpool upgrade -v

## upgrade all pools
zpool upgrade -a

## upgrade specific pool, use “zpool get all <pool>” to obtain version number of a pool
zpool upgrade data01

## upgrade to a specific version
zpool upgrade -V 10 data01

displaying zfs list

## list different types
zfs list -t filesystem
zfs list -t snapshot
zfs list -t volume

zfs list -t all -r <zpool>

## recursive display
zfs list -r data01/oracle

## complex listing
zfs list -o name,mounted,sharenfs,mountpoint

Note: there are a number of attributes that you can use in a complex listing, so use the man page to see them all

creating ## persuming i have a pool called data01 create a /data01/apache filesystem
zfs create data01/apache

## using a different mountpoint
zfs create -o mountpoint=/oracle data01/oracle

## create a volume – the device can be accessed via /dev/zvol/[rdsk|dsk]/data03/swap
zfs create -V 50mb data01/swap
swap -a /dev/zvol/dsk/data01/swap

Note: don’t use a zfs volume as a dump device it is not supported

destroying zfs destroy data01/oracle

## using the recusive options -r = all children, -R = all dependants
zfs destroy -r data01/oracle
zfs destroy -R data01/oracle

mounting zfs mount data01

# you can create temporary mount that expires after unmounting
zfs mount -o mountpoint=/tmpmnt data01/oracle

Note: there are all the normal mount options that you can apply i.e ro/rw, setuid

unmounting zfs umount data01
share zfs share data01

## Persist over reboots
zfs set sharenfs=on data01

## specific hosts
zfs set sharenfs=”rw=@″ data01/apache

unshare zfs unshare data01

## persist over reboots
zfs set sharenfs=off data01

snapshotting ## snapshotting is like taking a picture, delta changes are recorded to the snapshot when the original file system changes, to
## remove a dataset all previous snaphots have to be removed, you can also rename snapshots.
## You cannot destroy a snapshot if it has a clone

## creating a snapshot
zfs snapshot data01@10022010

## renaming a snapshot
zfs snapshot data01@10022010 data01@keep_this

## destroying a snapshot
zfs destroy data01@10022010

rollback ## by default you can only rollback to the lastest snapshot, to rollback to older one you must delete all newer snapshots
zfs rollback data01@10022010
cloning/promoting ## clones are writeable filesystems that was upgraded from a snapshot, a dependency will remain on the snapshot as long as the
## clone exists. A clone uses the data from the snapshot to exist. As you use the clone it uses space separate from the snapshot.

## clones cannot be created across zpools, you need to use send/receive see below topics

## cloning
zfs clone data01@10022010 data03/clone
zfs clone -o mountpoint=/clone data01@10022010 data03/clone

## promoting a clone, this allows you to destroy the original file ssytem that the clone is attached to
zfs promote data03/clone

Note: the clone must reside in the same pool

renaming ## the dataset must be kept within the same pool
zfs rename data03/ora_disk01 data03/ora_d01

Note: you have two options
-p creates all the non-existing parent datasets
-r recursively rename the sanpshots of all descendent datasets (used with snapshots only)

Compression ## You enable compression by seeting a feature, compressions are on, off, lzjb, gzip, gzip[1-9] ans zle, not that it only start
## compression when you turn it on, other existing data will not be compressed
zfs set compression=lzjb data03/apache

## you can get the compression ratio
zfs get compressratio data03/apache

Deduplication ## you can save disk space using deduplication which can be on file, block or byte, for example using file each file is hashed with a
## cryptographic hashing algorithm such as SHA-256, if a file matches then we just point to the existing file rather than storing a
## new file, this is ideal for small files but for large files a single character change would mean that all the data has to be copied

## block deduplication allows you to share all the same blocks in a file minus the blocks that are different, this allows to share the
## unique blocks on disk and the reference shared blocks in RAM, however it may need a lot of RAM to keep track of which blocks
## are shared and which are not., however this is the preferred option other than file or byte deduplication. Shared blocks are
## stored in what is called a “deduplication table”, the more deduplicated blocks the larger the table, the table is read everytime
## to make a block change thus the table should be held in fast RAM, if you run out of RAM then the table will spillover onto disk.

## So how much RAM do you need, you can use the zdb command to check, take the “bp count”, it takes about 320 bytes of ram
## for each deduplicate block in the pool, so in my case 288674 means I would need about 92MB, for example a 200GB would need
## about 670MB for the table, a good rule would be to allow 5GB of RAM for every 1TB of disk.

## to see the block the dataset consumes
zdb -b data01

## to turn on deduplicate
zfs set dedup=on data01/text_files

## to see the deduplicatio ratio
zfs get dedupratio data01/text_files

## to see the histrogram of howm many blocks are referenced how many time
zdb -DD <pool>

getting parameters ## List all the properties
zfs get all data03/oracle

## get a specific property
zfs get setuid data03/oracle

## get a list of a specific properites for all datasets
zfs get compression

Note: the source column denotes if the value has been change from it default value, a dash in this column means it is a read-only value

setting parameters ## set and unset a quota
zfs set quota=50M data03/oracle
zfs set quota=none data03/oracle

Note: use the command “zfs get all <dataset> ” to obtain list of current settings

inherit ## set back to the default value
zfs inherit compression data03/oracle
upgrade ## List the upgrade paths
zfs upgrade -v

## List all the datasets that are not at the current level
zfs upgrade

## upgrade a specific dataset
upgrade -V <version> data03/oracle

send/receive ## here is a complete example of a send and receive with incremental update

## create some test files
mkfile -v 100m /zfs/master
mkdir -v 100m /zfs/slave

## create mountpoints
mkdir /master
mkdir /slave

## Create the pools
zpool create master
zpool create slave

## create the data filesystem
zfs create master/data

## create a test file
echo “created: 09:58” > /master/data/test.txt

## create a snapshot and send it to the slave, you could use SSH or tape to transfer to another server (see below)
zfs snapshot master/data@1
zfs send master/data@1 | zfs receive slave/data

## set the slave to read-only because you can cause data corruption, make sure if do this before accessing anything the
## slave/data directory
zfs set readonly=on slave/data

## update the original test.txt file
echo “`date`” >> /master/data/text.txt

## create a second snapshot and send the differences, you may get an error message saying that the desination had been
## modified this is because you did not set the slave/data to ready only (see above)
zfs snapshot master/data@2
zfs send -i master/data@1 master/data@2 | zfs receive slave/data


## using SSH
zfs send master/data@1 | ssh backup_server zfs receive backups/data@1

## using a tape drive, you can also use cpio
zfs send master/data@1 > /dev/rmt/0
zfs receive slave/data2@1 < /dev/rmt/0
zfs rename slave/data slave/data.old
zfs rename slave/data2 slave/data

## you can also save incremental data
zfs send master/data@12022010 > /dev/rmt/0
zfs send -i master/data@12022010 master/data@13022010 > /dev/rmt/0

## Using gzip to compress the snapshot
zfs send master/fs@snap | gzip > /dev/rmt/0

allow/unallow ## display the permissions set and any user permissions
zfs allow master

## create a permission set
zfs allow -s @permset1 create,mount,snapshot,clone,promote master

## delete a permission set
zfs unallow -s @permset1 master

## grant a user permissions
zfs allow vallep @permset1 master

## revoke a user permissions
zfs unallow vallep @permset1 master

Note: there are many permissions that you can set so see the man page or just use the “zfs allow” command

Quota/Reservation ## Not strickly a command but wanted to discuss here, you can apply a quota to a dataset, you can reduce this quota only if the
## quota has not already exceeded, if you exceed the quota you will get a error message, you also have reservations which will
## guarantee that a specified amount of disk space is available to the filesystem, both are applied to datasets and there
## descendants (snapshots, clones)

## Newer versions of Solaris allow you to set group and user quota’s

## you can also use refquota and refreservation to manage the space without accounting for disk space consumed by descendants
## such as snapshots and clones. Generally you would set quota and reservation higher than refquota and refreservation

  • quota & reservation – properties are used for managing disk space consumed by datasets and their descendants
  • refquota & refreservation – properties are used for managing disk space consumed by datasets only

## set a quota
zfs set quota=100M data01/apache

## get a quota
zfs get quota data01/apache

## setup user quota (use groupquota for groups)
zfs set userquota@vallep=100M data01/apache

## remove a user quota (use groupquota for groups)
zfs set userquota@vallep=none data01/apache

## List user quota (use groupspace for groups), you can alsolist users with quota’s for exampe root user
zfs userspace data01/apache
zfs get userused@vallep data01/apache

ZFS tasks
Replace failed disk # List the zpools and identify the failed disk
zpool list

# replace the disk (can use same disk or new disk)
zpool replace data01 c1t0d0
zpool replace data01 c1t0d0 c1t1d0

# clear any existing errors
zpool clear data01

# scrub the pool to check for anymore errors (this depends on the size of the zpool as it can take a long time to complete
zpool scrub data01

# you can now remove the failed disk in the normal way depending on your hardware

Expand a pools capacity # you cannot remove a disk from a pool but you can replace it with a larger disk
zpool replace data01 c1t0d0 c2t0d0
zpool set autoexpand=on data01
Install the boot block # the command depends if you are using a sparc or a x86 system
sparc – installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t1d0
x86    – installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t1d0s0
Lost root password # You have two options to recover the root password

## option one
ok> boot -F failsafe
whne requested follow the instructions to mount the rpool on /a
cd /a/etc
vi passwd|shadow
init 6

## Option two
ok boot cdrom|net -s (you can boot from the network or cdroml)
zpool import -R /a rpool
zfs mount rpool/ROOT/zfsBE
cd /a/etc
vi passwd|shadow
init 6

Primary mirror disk in root is unavailable or fails # boot the secondary mirror
ok> boot disk1

## offline and unconfigure failed disk, there may be different options on unconfiguring a disk depends on the hardware
zpool offline rpool c0t0d0s0
cfgadm -c unconfigure c1::dsk/c0t0d0

# Now you can physically replace the disk, reconfigure it and bring it online
cfgadm -c configure c1::dsk/c0t0d0
zpool online rpool c0t0d0

# Let the pool know you have replaced the disk
zpool replace rpool c0t0d0s0

# if the replace above fails the detach and reattach the primary mirror
zpool deatch rpool c0t0d0s0
zpool attach rpool c0t1d0s0 c0t0d0s0

# make checks
zpool status rpool

# dont forget to add the boot block (see above)

Resize swap area (and dump areas) # You can resize the swap if it is not being used, first record the size and if it is being used
swap -l

# resize the swap area, first by removing it
swap -d /dev/zvol/dsk/rpool/swap
zpool set volsize=2G rpool/swap

# Now activate the swap and check the size, if the -a option does not work then use “swapadd” command
swap -a /dev/zvol/dsk/rpool/swap
swap -l

Note: if you cannot delete the original swap area due to being too busy then simple add another swap area, the same procedure is used for dump areas but using the “dumpadm” command

From 32 to 2 ports: Ideal SATA/SAS Controllers for ZFS & Linux MD RAID


I need a lot of reliable and cheap storage space (media collection, backups). Hardware RAID tends to be expensive and clunky. I recognize quite a few advantages in ZFS on Solaris/FreeBSD, and Linux MD RAID:

  • Performance. In many cases they are as fast as hardware RAID, and sometimes faster because the OS is aware of the RAID layout and can optimize I/O patterns for it. Indeed, even the most compute intensive RAID5 or 6 parity calculations take negligible CPU time on a modern processor. For a concrete example, Linux 2.6.32 on a Phenom II X4 945 3.0GHz computes RAID6 parity at close to 8 GB/s on a single core (check dmesg: “raid6: using algorithm sse2x4 (7976 MB/s)”). So achieving a throughput of 500 MB/s on a Linux MD raid6 array requires spending less than 1.5% CPU time computing parity. Now regarding the optimized I/O patterns, here is an interesting anecdote: one of the steps that Youtube took in its early days to scale their infrastructure up was to switch from hardware RAID to software RAID on their database server. They noticed a 20-30% increase in I/O throughput. WatchSeattle Conference on Scalability: YouTube Scalability @ 34’50”.
  • Scalability. ZFS and Linux MD RAID allow building arrays across multiple disk controllers, or multiple SAN devices, alleviating throughput bottlenecks that can arise on PCIe links, or GbE links. Whereas hardware RAID is restricted to a single controller, with no room for expansion.
  • Reliability. No hardware RAID = one less hardware component that can fail.
  • Ease of recoverability. The data can be recovered by putting the disks in any server. There is no reliance on a particular model of RAID controller.
  • Flexibility. It is possible to create arrays on any disk on any type of controller in the system, or to move disks from one controller to another.
  • Ease of administration. There is only one software interface to learn: zpool(1M) or mdadm(8). No need to install proprietary vendor tools, or to reboot into BIOSes to manage arrays.
  • Cost. Obviously cheaper since there is no hardware RAID controller to buy.

Consequently, many ZFS and Linux MD RAID users, such as me, look for non-RAID controllers that are simply reliable, fast, cheap, and otherwise come with no bells and whistles. Most motherboards have up to 4 or 6 onboard ports (be sure to always enable AHCI mode in the BIOS as it is the best designed hardware interface that a chip can present to the OS to enable maximum performance), but for more than 4 or 6 disks, there are surprisingly not that many choices of controllers. Over the years, I have spent quite some time on the controllers manufacturers’ websites, the LKML, linux-ide and ZFS mailing lists, and have established a list of SATA/SAS controllers that are ideal for ZFS or Linux MD RAID. I also included links to online retailers because some of these controllers are not that easy to find online.

The reason the list contains SAS controllers is because they are just as good as an option as SATA controllers: many of them are as inexpensive as SATA controllers (even though they target the enterprise market), they are fully compatible with SATA 3Gbps and 6Gbps disks, and they support all the usual features: hotplug, queueing, etc. A SAS controller typically present SFF-8087 connectors, also known as internal mini SAS, or even iPASS connectors. Up to 4 SATA drives can be connected to such a connector with an SFF-8087 to 4xSATA forward breakout cable (as opposed to reverse breakout). This type of cable usually sells for $15-30. Here are a few links if you have trouble finding them.

There are really only 4 significant manufacturers of discrete non-RAID SATA/SAS controller chips on the market: LSI, Marvell, JMicron, and Silicon Image. Controller cards from Adaptec, Areca, HighPoint, Intel, Supermicro, Tyan, etc, most often use chips from one of these 4 manufacturers.

Here is my list of non-RAID SATA/SAS controllers, from 16-port to 2-port controllers, with the kernel driver used to support them under Linux, and Solaris. There is also limited information on FreeBSD support. I focused on native PCIe controllers only, with very few PCI-X (actually only 1 very popular: 88SX6081). The MB/s/port number in square brackets indicates the maximum practical throughput that can be expected from each SATA port, assuming concurrent I/O on all ports, given the bottleneck of the host link or bus (PCIe or PCI-X). I assumed for all PCIe controllers that only 60-70% of the maximum theoretical PCIe throughput can be achieved, and for all PCI-X controllers that only 80% of the maximum theoretical PCI-X throughput can be achieved on this bus. These assumptions concur with what I have seen in real world benchmarks assuming a Max_Payload_Size setting of either 128 or 256 bytes for PCIe (a common default value), and a more or less default PCI latency timer setting for PCI-X. As of May 2010, modern disks can easily reach 120-130MB/s of sequential throughput at the beginning of the platter, so avoid controllers with a throughput of less than 150MB/s/port if you want to reduce the possibility of bottlenecks to zero.

32 ports

  • [SAS] 4 x switched Marvell 88SE9485, 6Gbps, PCIe (gen2) x16 [150-175MB/s/port]
    [Update 2011-09-29: Availability: $850 $850. This is a HighPoint HBA combining 4 x 8-port Marvell 88SE9485 with PCIe switching technology: RocketRAID 2782.]

    • Linux/Solaris/FreeBSD support: see Marvell 88SE9485 or 88SE9480 below

24 ports

  • [SAS] 3 x switched Marvell 88SE9485, 6Gbps, PCIe (gen2) x16 [200-233MB/s/port]
    [Update 2011-09-29: Availability: $540 $620. This is a HighPoint HBA combining 3 x 8-port Marvell 88SE9485 with PCIe switching technology: RocketRAID 2760A.]

    • Linux/Solaris/FreeBSD support: see Marvell 88SE9485 or 88SE9480 below

16 ports

  • [SAS] LSI SAS2116, 6Gbps, PCIe (gen2) x8 [150-175MB/s/port]
    Availability: $400 $510. LSI HBA based on this chip: LSISAS9200-16e, LSISAS9201-16i. [Update 2010-10-27: only the model with external ports used to be available but now the one with internal ports is available and less expensive.]

  • [SAS] 2 x switched Marvell 88SE9485, 6Gbps, PCIe (gen2) x16 [300-350MB/s/port]
    [Update 2011-09-29: Availability: $450 $480. This is a HighPoint HBA combining 3 x 8-port Marvell 88SE9485 with PCIe switching technology: RocketRAID 2740 and 2744.]

    • Linux/Solaris/FreeBSD support: see Marvell 88SE9485 or 88SE9480 below

8 ports

  • [SAS] Marvell 88SE9485 or 88SE9480, 6Gbps, PCIe (gen2) x8 [300-350MB/s/port]
    Availability: $280. [Update 2011-07-01: Supermicro HBA based on this chip: AOC-SAS2LP-MV8]. Areca HBA based on the 9480: ARC-1320. HighPoint HBA based on the 9485: RocketRAID 272x. Lots of bandwidth available to each port. However it is currently not supported by Solaris. I would recommend the LSI SAS2008 instead, which is cheaper, better supported, and provides just as much bandwidth.

    • Linux support: mvsas (94xx: 2.6.31+, ARC-1320: 2.6.32+)
    • Solaris support: not supported (see 88SE6480)
    • Mac OS X support: [Update 2014-06-26: the only 88SE9485 or 88SE9480 HBAs supported by Mountain Lion (10.8) and up seem to be HighPoint HBAs]
  • [SAS] LSI SAS2008, 6Gbps, PCIe (gen2) x8 [300-350MB/s/port]
    Availability: $130 $140 $180 $220 $220 $230 $240 $290. [Update 2010-12-21: Intel HBA based on this chip: RS2WC080]. Supermicro HBAs based on this chip: AOC-USAS2-L8i AOC-USAS2-L8e (these are 2 “UIO” cards with the electronic components mounted on the other side of the PCB which may not be mechanically compatible with all chassis). LSI HBAs based on this chip: LSISAS9200-8e LSISAS9210-8i LSISAS9211-8i LSISAS9212-4i4e. Lots of bandwidth per port. Good Linux and Solaris support.

  • [SAS] LSI SAS1068E, 3Gbps, PCIe (gen1) x8 [150-175MB/s/port]
    Availability: $110 $120 $150 $150. Intel HBAs based on this chip: SASUC8I. Supermicro HBAs based on this chip: AOC-USAS-L8i AOC-USASLP-L8i (these are 2 “UIO” cards – see warning above.) LSI HBAs based on this chip: LSISAS3081E-R LSISAS3801E. Can provide 150-175MB/s/port of concurrent I/O, which is good enough for HDDs (but not SSDs). Good Linux and Solaris support. This chip is popular because it has very good Solaris support and was chosen by Sun for their second generation Sun Fire X4540 Server “Thumper”. However, beware, this chip does not support drives larger than 2TB.

    • Linux support: mptsas
    • Solaris support: mpt
    • FreeBSD support: mpt (supported at least since 7.3)
  • [SATA] Marvell 88SX6081, 3Gbps, PCI-X 64-bit 133MHz [107MB/s/port]
    Availability: $100. Supermicro HBAs based on this chip: AOC-SAT2-MV8 Based on PCI-X, which is an aging technology being replaced with PCIe. The approximate 107MB/s/port of concurrent I/O it supports is a bottleneck with modern HDDs. However this chip is especially popular because it has very good Solaris support and was chosen by Sun for their first generation Sun Fire X4500 Server “Thumper”.

    • Linux support: sata_mv (no suspend support)
    • Solaris support: marvell88sx
    • FreeBSD support: ata (supported at least since 7.0, if the hptrr driver is commented out)
  • [SAS] Marvell 88SE6485 or 88SE6480, 3Gbps, PCIe (gen1) x4 [75-88MB/s/port]
    Availability: $100. Supermicro HBAs based on this chip: AOC-SASLP-MV8. The PCIe x4 link is a bottleneck for 8 drives, restricting the concurrent I/O to 75-88MB/s/port. A better and slightly more expensive alternative is the LSI SAS1068E.

4 ports

  • [SAS] LSI SAS2004, 6Gbps, PCIe (gen2) x4 [300-350MB/s/port]
    Availability: $160. LSI HBA based on this chip: LSISAS9211-4i. Quite expensive; I would recommend buying a (cheaper!) 8-port controller.

  • [SAS] LSI SAS1064E, 3Gbps, PCIe (gen1) x8 [300-350MB/s/port]
    Availability: $120 $130. Intel HBA based on this chip: SASWT4I. [Update 2010-10-27: LSI HBA based on this chip: LSISAS3041E-R.] It is quite expensive. [Update 2014-12-04: And it does not support drives larger than 2TB.] For these reasons, I recommend instead buying a cheaper 8-port controller.

    • Linux support: mptsas
    • Solaris support: mpt
    • FreeBSD support: mpt (supported at least since 7.3)
  • [SAS] Marvell 88SE6445 or 88SE6440, 3Gbps, PCIe (gen1) x4 [150-175MB/s/port]
    Availability: $80. Areca HBA based on the 6440: ARC-1300. Adaptec HBA based on the 6440: ASC-1045/1405. Provides good bandwidth at a decent price.

    • Linux support: mvsas (6445: 2.6.25 or 2.6.31 ?, 6440: 2.6.25+, ARC-1300:2.6.32+)
    • Solaris support: not supported (see 88SE6480)
  • [SATA] Marvell 88SX7042, 3Gbps, PCIe (gen1) x4 [150-175MB/s/port]
    Availability: $70. Adaptec HBA based on this chip: AAR-1430SA. Rosewill HBA based on this chip: RC-218. This is the only 4-port SATA controller supported by Linux providing acceptable throughput to each port. [2010-05-30 update: I bought one for $50 from Newegg in October 2009. Listed at $70 when I wrote this blog. Currently out of stock and listed at $90. Its popularity is spreading…]

  • [SAS] Marvell 88SE6340, 3Gbps, PCIe (gen1) x1 [38-44MB/s/port]
    Hard to find. Only found references to this chip on Marvell’s website. Performance is low anyway (38-44MB/s/port).

    • Linux support: mvsas
    • Solaris support: not supported (see 88SE6480)
  • [SATA] Marvell 88SE6145 or 88SE6141, 3Gbps, PCIe (gen1) x1 [38-44MB/s/port]
    Hard to find. Chip seems to be mostly found on motherboards for onboard SATA. Performance is low anyway (38-44MB/s/port).

    • Linux support: ahci
    • Solaris support: ahci
    • FreeBSD support: ahci

2 ports

  • [SATA] Marvell 88SE9128 or 88SE9125 or 88SE9120, 6Gbps, PCIe (gen2) x1 [150-175MB/s/port]
    Availability: $25 $35. HighPoint HBA based on this chip: Rocket 620. LyCOM HBA based on this chip: PE-115. Koutech HBA based on this chip: PESA230. This is the only 2-port chip on the market with no bottleneck caused by the PCIe link at Max_Payload_Size=128. Pretty surprising that it is being sold for such a low price.

    • Linux support: ahci
    • Solaris support: not supported [Update 2010-09-21: Despite being AHCI-compliant, this series of chips seems unsupported by Solaris according to reader comments, see below.]
    • FreeBSD support: ahci
  • [SATA] Marvell 88SE6121, 3Gbps, PCIe (gen1) x1 [75-88MB/s/port]
    Hard to find. Chip seems to be mostly found on motherboards for onboard SATA.

    • Linux support: ahci
    • Solaris support: ahci
    • FreeBSD support: ahci
  • [SATA] JMicron JMB362 or JMB363 or JMB366, 3Gbps, PCIe (gen1) x1 [75-88MB/s/port]
    Availability: $22.

    • Linux support: ahci
    • Solaris support: ahci
    • FreeBSD support: ahci
  • [SATA] SiI3132, 3Gbps, PCIe (gen1) x1 [75-88MB/s/port]
    Availability: $20. Warning: the overall bottleneck of the PCIe link is 150-175MB/s, or 75-88MB/s/port, but the chip has a 110-120MB/s bottleneck per port. So a single SATA device on a single port cannot fully use the 150-175MB/s by itself, it will be bottlenecked at 110-120MB/s.

Finding cards based on these controller chips can be surprisingly difficult (I have had to zoom on product images on newegg.com to read the inscription on the chip before buying), hence the reason I included some links to online retailers.

For reference, the maximum practical throughputs per port I assumed have been computed with these formulas:

  • For PCIe gen2: 300-350MB/s (60-70% of 500MB/s) * pcie-link-width / number-of-ports
  • For PCIe gen1: 150-175MB/s (60-70% of 250MB/s) * pcie-link-width / number-of-ports
  • For PCI-X 64-bit 133MHz: 853MB/s (80% of 1066MB/s) / number-of-ports

To anyone building ZFS or Linux MD RAID storage servers, I recommend to first make use of all onboard AHCI ports on the motherboard. Then put any extra disks on a discrete controller, and I recommend specifically these ones:

  • For a 2-port controller: Marvell 88SE9128 or 88SE9125 or 88SE9120. I do not primarily recommend it because it is SATA 6Gbps, but because it supports PCIe gen2, which allows the controller to handle an overall throughput of at least 300-350MB/s, or 150-175MB/s/port, with a default PCIe Max_Payload_Size setting of 128 bytes. It is also fully AHCI compliant, in other words robust, well-designed, and virtually compatible with all operating systems; a notable exception is Solarisfor which I recommend instead the next best controller: JMicron JMB362 or JMB363 or JMB366. The icing on the cake is that cards using these chips are inexpensive (starting from $22, or $11/port).
  • For an 8-port controller: LSI SAS1068E if you are fine with it only supporting drives up to 2TB. Controllers based on this chip can be found inexpensively (starting from $110, or $13.75/port) and are supported out of the box by many current and older Linux and Solaris versions. In fact this chip is the one that Sun used in their second generation Sun Fire X4540 Server “Thumper”. The fact that it can support up to 150-175MB/s/port due to the PCIe bottleneck with concurrent I/O on all ports is sufficient for current disks. However if you need more throughput (eg. are using SSDs), or need to use drives larger than 2TB, then go for its more expensive successor, LSI SAS2008, which supports PCI gen2, which should allow for 300-350MB/s/port before hitting the PCIe bottleneck.

NAS4FREE with SAMBA 4 in Active Directory – fun with ACLs


ACL set codes

Working with permissions on NAS4FREE shell is pleasant and enjoyable experience …. *cough*, *cough*.

$ getfacl /tank/shared

# file: /tank/shared
# owner: aduser
# group: 500

full_set = rwxpdDaARWcCos = all permissions
modify_set = rwxpdDaARWc–s = all permissions except write_acl, write_owner
read_set = r—–a-R-c— = read_data, read_attributes, read_xattr, read_acl
write_set = -w-p—A-W—- = write_data, append_data, write_attributes, write_xattr
NFSv4 ACL legend (read from top, down and exit on first match)

           (r)read data +|||||||||||||:||||||+ (I)nherited
           (w)rite data -+||||||||||||:|||||+- (F)ailed access (audit)
              e(x)ecute --+|||||||||||:||||+-- (S)uccess access (audit)
               a(p)pend ---+||||||||||:|||+--- (n)o propagate
               (d)elete ----+|||||||||:||+---- (i)nherit only
         (D)elete child -----+||||||||:|+----- (d)irectory inherit
          read (a)ttrib ------+|||||||:+------ (f)ile inherit
         write (A)ttrib -------+||||||
           (R)ead xattr --------+|||||
          (W)rite xattr ---------+||||
             read a(c)l ----------+|||
            write a(C)l -----------+||
         change (o)wner ------------+|
                   sync -------------+


Changing permissions over NAS4FREE console is possible

setfacl -m u:aduser:rwxpdaARWc:fd:allow folderHere
setfacl -m g:adgroup_shared_share_access:rwxpDaARWcCo:fd:allow shared/

But it’s probably easier to manage it from Windows workstation

Pick one responsible user and grant him/her SeDiskOperatorPrivilege

net rpc rights grant aduser SeDiskOperatorPrivilege -U adadmin
net rpc rights list accounts -P
net rpc rights list -P

Also add in NAS4FREE web interface under Services|CIFS/SMB|Share|Edit|Auxiliary parameters

admin users = aduser
valid users = @adgroup_shared_share_access, @"ADgroup Shared Share Access"
write list = @adgroup_shared_share_access, @"ADgroup Shared Share Access"

And just let him do the hard work of dealing with other users.

Useful links




A Complete Guide to FreeNAS Hardware Design, Part I: Purpose and Best Practices


A guide to selecting and building FreeNAS hardware, written by the FreeNAS Team, is long past overdue by now. For that, we apologize. The issue was the depth and complexity of the subject, as you’ll see by the extensive nature of this four part guide, due to the variety of ways FreeNAS can be utilized. There is no “one-size-fits-all” hardware recipe. Instead, there is a wealth of hardware available, with various levels of compatibility with FreeNAS, and there are many things to take into account beyond the basic components, from use case and application to performance, reliability, redundancy, capacity, budget, need for support, etc. This document draws on years of experience with FreeNAS, ZFS, and the OS that lives underneath FreeNAS, FreeBSD. Its purpose is to give guidance on intelligently selecting hardware for use with the FreeNAS storage operating system, taking the complexity of its myriad uses into account, as well as providing some insight into both pathological and optimal configurations for ZFS and FreeNAS.freenashome

A word about software defined storage:

FreeNAS is an implementation of Software Defined Storage; although software and hardware are both required to create a functional system, they are decoupled from one another. We develop and provide the software and leave the hardware selection to the user. Implied in this model is the fact that there are a lot of moving pieces in a storage device (figuratively, not literally). Although these parts are all supposed to work together, the reality is that all parts have firmware, many devices require drivers, and the potential for there to be subtle (or gross) incompatibilities is always present.

Best Practices

ECC RAM or Not?

This is probably the most contested issue surrounding ZFS (the filesystem that FreeNAS uses to store your data) today. I’ve run ZFS with ECC RAM and I’ve run it without. I’ve been involved in the FreeNAS community for many years and have seen people argue that ECC is required and others argue that it is a pointless waste of money. ZFS does something no other filesystem you’ll have available to you does: it checksums your data, and it checksums the metadata used by ZFS, and it checksums the checksums. If your data is corrupted in memory before it is written, ZFS will happily write (and checksum) the corrupted data. Additionally, ZFS has no pre-mount consistency checker or tool that can repair filesystem damage. This is very nice when dealing with large storage arrays as a 64TB pool can be mounted in seconds, even after a bad shutdown. However if a non-ECC memory module goes haywire, it can cause irreparable damage to your ZFS pool that can cause complete loss of the storage. For this reason, I highly recommend the use of ECC RAM with “mission-critical” ZFS. Systems with ECC RAM will correct single bit errors on the fly, and will halt the system before they can do any damage to the array if multiple bit errors are detected. If it’s imperative that your ZFS based system must always be available, ECC RAM is a requirement. If it’s only some level of annoying (slightly, moderately…) that you need to restore your ZFS system from backups, non-ECC RAM will fit the bill.

How Much RAM is needed?

FreeNAS requires 8 GB of RAM for the base configuration. If you are using plugins and/or jails, 12 GB is a better starting point. There’s a lot of advice about how RAM hungry ZFS is, how it requires massive amounts of RAM, an oft quoted number is 1GB RAM per TB of storage. The reality is, it’s complicated. ZFS does require a base level of RAM to be stable, and the amount of RAM it needs to be stable does grow with the size of the storage. 8GB of RAM will get you through the 24TB range. Beyond that 16GB is a safer minimum, and once you get past 100TB of storage, 32GB is recommended. However, that’s just to satisfy the stability side of things. ZFS performance lives and dies by its caching. There are no good guidelines for how much cache a given storage size with a given number of simultaneous users will need. You can have a 2TB array with 3 users that needs 1GB of cache, and a 500TB array with 50 users that need 8GB of cache. Neither of those scenarios are likely, but they are possible. The optimal cache size for an array tends to increase with the size of the array, but outside of that guidance, the only thing we can recommend is to measure and observe as you go. FreeNAS includes tools in the GUI and the command line to see cache utilization. If your cache hit ratio is below 90%, you will see performance improvements by adding cache to the system in the form of RAM or SSD L2ARC (dedicated read cache devices in the pool).

RAID vs. Host Bus Adapters (HBAs)

ZFS wants direct control of the underlying storage that it is putting your data on. Nothing will make ZFS more unstable than something manipulating bits underneath ZFS. Therefore, connecting your drives to an HBA or directly to the ports on the motherboard is preferable to using a RAID controller; fortunately, HBAs are cheaper than RAID controllers to boot! If you must use a RAID controller, disable all write caching on it and disable all consistency checks. If the RAID controller has a passthrough or JBOD mode, use it. RAID controllers will complicate disk replacement and improperly configuring them can jeopardize the integrity of your volume (Using the write cache on a RAID controller is an almost sure-fire way to cause data loss with ZFS, to the tune of losing the entire pool).

Virtualization vs. Bare Metal

FreeBSD (the underlying OS of FreeNAS) is not the best virtualization guest: it lacks some virtio drivers, it lacks some OS features that make it a better behaved guest, and most importantly, it lacks full support from some virtualization vendors. In addition, ZFS wants direct access to your storage hardware. Many virtualization solutions only support hardware RAID locally (I’m looking at you, VMware) thus leading to enabling a worst case scenario of passing through a virtual disk on a datastore backed by a hardware RAID controller to a VM running FreeNAS. This puts two layers between ZFS and your data, one for the Host Virtualization’s filesystem on the datastore and another on the RAID controller. If you can do PCI passthrough of an HBA to a FreeNAS VM, and get all the moving pieces to work properly, you can successfully virtualize FreeNAS. We even include the guest VM tools in FreeNAS for VMware, mainly because we use VMware to do a lot of FreeNAS development. However if you have problems, there are no developer assets running FreeNAS as a production VM and help will be hard to come by. For this reason, I highly recommend that FreeNAS be run “On the Metal” as the only OS on dedicated hardware.

Josh Paetzel
iXsystems Director of IT

Part 2/4 of A Complete Guide to FreeNAS Hardware Design: Hardware Specifics >>

Build your own vCenter and vCloud Vagrant boxes


After releasing vagrant-vcenter the other day (see the blog post here), I decided to put the finishing touches on aPacker plugin I’ve been working on for quite some time.

First of all, what is Packer? As its website says, Packer is a tool for creating identical machine images for multiple platforms from a single source configuration. I use Packer extensively to build Vagrant boxes and templates for my homelab and I decided to build a post-processor plugin to create boxes in the format used by our Vagrant providers, please note that Packer and its post-processors are written in Go (also known as golang) that is completely new to me so the code may look “fugly” :-).

I created two new repositories over at the GoSDDC GitHub page, one contains the Packer post processor:https://github.com/gosddc/packer-post-processor-vagrant-vmware-ovf and another contains the Packer templates I use to build Vagrant boxes (currently Ubuntu and CentOS) https://github.com/gosddc/packer-templates .

Go is a compiled language, hence I’m distributing binaries as well, if you’re not comfortable building Go from source, please head over to the release page for packer-post-processor-vagrant-vmware-ovf and grab the latest release compiled for your platform (currently building for darwin (Mac OS X), Linux, FreeBSD, OpenBSD, NetBSD and Windows).

VNS3:net Network security & control on your terms


Security & connectivity for hybrid cloud deployments

Secure your vital data in motion and better monitor your network topology. Control a secure, scalable, highly available, meshed network across multiple clouds.  Create a logical group of resources in multiple cloud regions, zones, and providers.

Build enhanced network services over top of any cloud IaaS network by using VNS3’s layer 4-7 services and our plugin container ecosystem. Customers use VNS3 to do everything from connecting a flexible VPN IPsec tunnel to managing complex cross-cloud networks.

VNS3:net gives you cloud mobility and agility without compromises. Since its launch in 2008, VNS3:net has been used to secure over 800 million virtual device hours in any virtualized environment.


A software-only virtual appliance for any cloud

VNS3:net is different from other security and network routing solutions because it lets you create a layer 4-7 overlay network on top of your underlying network backbone. Secure all data in motion with encrypted VPNs and IPsec while adding flexibility to your application

VNS3:net is a hybrid virtual device that act as six devices in one:

  • router,
  • switch,
  • SSL/IPSec VPN concentrator,
  • firewall,
  • protocol re-distributor, and
  • extensible NFV

VNS3:net is a software-only virtual machine that integrates with existing network equipment and can be delivered as part of the application deployment in most virtualized infrastructures.

Meet compliance requirements, attest to data security, and manage your cloud deployments with VNS3:net. Dynamically launch and configure your overlay network in minutes using the REST API or web-based UI. Use VNS3:vpn for free, as a pay-as-you-go virtual network server to eliminates cloud networking risk.

Dramatically Simpler Cloud Security: Azure Security Center and Trend Micro

Dramatically Simpler Cloud Security: Azure Security Center and Trend Micro


Read about how Deep Security and Microsoft Azure Security Center simplify cloud security.

Written by: Trend Micro   March 23, 2016

Everyone seems to be looking for ways to simplify security, especially cloud security. That’s why Trend Micro has designed Azure security solutions to be as automatic, agile and flexible as Microsoft Azure itself. So it was no surprise when Microsoft chose to integrate the new Azure Security Center with Trend Micro. The Azure Security Center provides you with a single view of all your Azure subscriptions, performs live monitoring on workloads, and recommends remediation steps to address identified security issues. When the Azure Security Center discovers malware vulnerabilities on virtual machines it presents Trend Micro Deep Security as a recommended solution to address them. To make things even easier, you can then deploy Deep Security directly from the Azure Marketplace to enhance the protection of your environments from network attacks.

Why is Trend Micro presented as a recommended solution?
Trend Micro is a leader in cloud security1. And we have earned that position because we understand that to fully embrace the cloud, you need security that preserves its economic and operational benefits. Trend Micro has been working closely with Microsoft to ensure that we deliver elastic, flexible, and scalable security solutions that are compatible with the Azure environment. As a result, the integration of Trend Micro with Azure Security Center makes it easier than ever to secure cloud workloads.

Easily identify virtual machine security issues
Trend Micro Deep Security provides the security capabilities you need to meet shared responsibility in the cloud. And we do it with the industry’s most complete set of security capabilities for Microsoft Azure.
With Deep Security, you can detect and remove malware in real-time, protect against known and unknown vulnerabilities, including zero-day attacks. You can also detect suspicious or malicious activity, including integrity monitoring required to meet compliance with key regulations, including PCI DSS 3.1, HIPAA, and others.

Fast deployment and automated management
Deep Security monitors your Azure environment, automatically recognizing and provisioning security to new instances. Plus, Deep Security automates repetitive, resource-intensive security tasks, such as provisioning and deprovisioning, to dramatically reduce operational cost and time. So you get the security you need to move sensitive workloads to Microsoft Azure without compromising its promise of automation and agility.
Microsoft Azure Security Center will be available in beta to all Azure customers globally on December 2, 2015. Commercial availability is expected in 2016. To learn more about how Trend Micro Deep Security secures Azure workloads and sign up for a 30-day free trial, please visitwww.trendmicro.com/azure.

How to Deploy the Barracuda NG Firewall on Microsoft Azure via PowerShell


For most advanced networking features in the Microsoft Azure Cloud, such as multiple network interfaces or reserved IP addresses for the Cloud Service, you must deploy the Barracuda NG Firewall via PowerShell. You can either enter the commands directly into the Azure PowerShell or combine the commandlets to a custom deployment script. Using a custom PowerShell script allows for rapid deployment and fast recovery in case of failure. The number of network interfaces depends on the Instance size:

  • Small –  One network interface.
  • Medium – One network interface.
  • Large – Up to two network interfaces.
  • Extra Large – Up to four network interfaces.

Microsoft Azure charges apply. For more information, see the Microsoft Azure Pricing Calculator[1].


Example Deployment Script

You can combine the PowerShell commandlets to customize the deployment of your Barracuda NG Firewall in the Microsoft Azure cloud. See below for an example deployment script. This script assumes that you already configured a Regional VNET, Reserved IP (optional) in the Azure cloud, and the Azure Account for Azure PowerShell on your client.

#If needed import Azure PSD file 
#Import-Module "C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd"

# Use Default System Proxy 
[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials

# Modify the variables below
$vmname = "BNG-MultiNIC"
$RootPassword = "secretpa$$word"
$instanceSize = "Large"
$cloudService = "BNG-CS"
$Location = "North Europe"
$storageAccount ="mystorageaccount"
#Leave empty is no reserved IP is used 
$reservedIPname = ""
$VNetName = "NG-VNET"
$Subnet1 = "Frontend"
$Subnet2 = "Backend" 
$NIC1IP = ""
$NIC2IP = ""
#Enter a VM Image name below to use a custom image. If left empty the latest image from the Azure Marketplace is used. 
$image = ""
$availabilitySetName ="BarracudaNGAVSet" 
$azureSubscriptionName = "Pay-As-You-Go"

function AskYesNo( $title, $question, $YesInfo, $NoInfo ) {
    $yes = New-Object System.Management.Automation.Host.ChoiceDescription "&Yes", $YesInfo
    $no = New-Object System.Management.Automation.Host.ChoiceDescription "&No", $NoInfo
    $options = [System.Management.Automation.Host.ChoiceDescription[]]($yes, $no)
    $result = $host.ui.PromptForChoice($title, $question, $options, 0)
    return $result

Write-Host -NoNewLine "This script will create a "
Write-Host -NoNewLine -ForegroundColor yellow "dual-NIC Barracuda NG Firewall"
Write-Host " instance in Azure"
Write-Host ""
Write-Host -NoNewLine "Vnet name: "
Write-Host -ForegroundColor yellow $VNetName
Write-Host -NoNewLine "NIC 1: "
Write-Host -NoNewLine -ForegroundColor yellow "$NIC1IP in $Subnet1"
Write-Host " (management)"
Write-Host -NoNewLine "NIC 2: "
Write-Host -ForegroundColor yellow "$NIC2IP in $Subnet2"
Write-Host -NoNewLine "Azure DC: "
Write-Host -ForegroundColor yellow $Location

if ($reservedIPName -ne "")
    Write-Host "Using the Existing Reserved IP address: $reservedIPName" 

$yesorno = AskYesNo 'Do you want to continue?' $warn 'aborting script' 'using existing VNET' 
    switch ( $yesorno ) {
        0 { "OK! Creating a new Barracuda NG Firewall VM." }
        1 { 
            "Got it :( Please correct variable values in script and rerun."

# Create storage if it doesn't exist yet
if(!(Test-AzureName -Storage $storageAccount))
    Write-Host "Creating Storage Account $storageAccount in $Location"
    New-AzureStorageAccount -StorageAccountName $storageAccount -Location $Location

if ($reservedIPName -ne "") 
$reservedIP = Get-AzureReservedIP -ReservedIPName $reservedIPName
Write-Host "Using Existing Reserved IP!"

# Set storage account as default storage 
Set-AzureSubscription -SubscriptionName $azureSubscriptionName -CurrentStorageAccountName $storageAccount 

# If no explicit image is defined get the latest Barracuda NG Firewall Azure Image available in the Azure Marketplace
if ( $image -eq "")
    $image = Get-AzureVMImage | where { $_.ImageFamily -Match "Barracuda NG Firewall*"} | sort PublishedDate -Descending | select -ExpandProperty ImageName -First 1
    Write-Host "Using Image from Azure Marketplace..."

# Create Azure VM 
$vm1 = New-AzureVMConfig -Name $vmname -InstanceSize $instanceSize -Image $image –AvailabilitySetName $availabilitySetName
Add-AzureProvisioningConfig -Linux -LinuxUser "azureuser" -Password $RootPassword -VM $vm1 -NoSSHEndpoint

# Add Endpoints for 1st NIC of the Barracuda NG Firewall 
Add-AzureEndpoint -Protocol tcp -LocalPort 22 -PublicPort 22 -Name "SSH" -VM $vm1
Add-AzureEndpoint -Protocol tcp -LocalPort 807 -PublicPort 807 -Name "MGMT" -VM $vm1
Add-AzureEndpoint -Protocol tcp -LocalPort 691 -PublicPort 691 -Name "TINATCP" -VM $vm1
Add-AzureEndpoint -Protocol udp -LocalPort 691 -PublicPort 691 -Name "TINAUDP" -VM $vm1
Write-Host "Added Endpoints..."

# Define Subnet and static IP Address for 1st NIC
Set-AzureSubnet -SubnetName $Subnet1 -VM $vm1 
Set-AzureStaticVNetIP -IPAddress $NIC1IP -VM $vm1 
Write-Host "Configured First NIC..."

# Add Additional NICS 
Add-AzureNetworkInterfaceConfig -Name "NIC2" -SubnetName $Subnet2 -StaticVNetIPAddress $NIC2IP -VM $vm1 
Write-Host "Added Second NIC..."

# Create Barracuda NG Firewall VM 
if ($reservedIPName -eq "") 
    New-AzureVM -ServiceName $cloudService -VM $vm1 -Location $Location -VNetName $VNetName 
    Write-Host "Creating VM without Reserved IP Address..."
    New-AzureVM -ServiceName $cloudService -VM $vm1 -ReservedIPName $reservedIPName -Location $Location -VNetName $VNetName 
    Write-Host "Creating VM with Reserved IP Address $reservedIPName... "
Write-Host "Script is done. Creating the Virtual Machine can take a while. Have a cup of coffee! Use Barracuda NG Admin to login to $cloudService.cloudapp.net: user: root, password: $RootPassword)"


In this article

Before You Begin

  • Create a Microsoft Azure account[2].
  • Download and install the latest version of Azure PowerShell[3].
  • Purchase a Barracuda NG Azure license or get a Barracuda NG Azure license from the Barracuda Networks Evaluation page[4]:
    1. From the Select a Product list, select Barracuda NG Firewall Azure under the Public Cloud Solutionscategory.
    2. From the Select Edition list, select the Level that you want. Azure Level 3 or 4 required for multi-NIC Deployments
    3. Complete and submit the rest of the form. You will receive an email containing your serial number and license token.

Step 1. Configure your Azure PowerShell to use your Azure Account

Import the Azure Subscription file, to associate Azure PowerShell with your Azure account.

  1. Open an Azure PowerShell.
  2. To download your publishsettingsfile, enter:
    1. GetAzurePublishSettingsFile
  3. The download popup of your browser opens. Save the file.
  4. Import the publishsettingsfile by entering:
    1. ImportAzurePublishSettingsFile PATH_TO_FILE
  5. Check your subscription by entering. If CurrentStorageAccountName is set, make sure that the storage account is in the same location you want to create the VM in.
    1. GetAzureSubscription


Step 2. Create an Azure Regional Virtual Network

You must use a Regional VNet to deploy the Barracuda NG Firewall. Older Affinitygroup-based VNets are not compatible with reserved static IP addresses, static internal IP addresses, Public IP Addresses (PIP), or multiple network interfaces. Configuration information of the VNet is stored in an XML file and then deployed in the Azure Cloud via PowerShell commandlet. An example vmnet.xml with 2 subnet:

<NetworkConfiguration xmlns="http://schemas.microsoft.com/ServiceHosting/2011/07/NetworkConfiguration">
      <VirtualNetworkSite name="NEVNET" Location="North Europe">
          <Subnet name="Frontend">
          <Subnet name="Backend">
  1. Open the Azure PowerShell.
  2. If VNets already exist, export the existing Virtual Networks to a xml file
    1. GetAzureVNetConfig ExportToFile c:\azure\vmnet.xml
  3. Edit the vnet.xml file and enter the configuration for your VIRTUALNETWORKSITE. Use the example file above as a guideline. If you are using multiple network interfaces, create one subnet per network interface.
  4. Upload the VNet configuration file:
    1. SetAzureVNetConfig ConfigurationPath PATHTOYOURVNETXMLFILE

The virtual network is now listed in VIRTUAL NETWORKS in the web UI, via PowerShell:

  1. GetAzureVNetSite VNetName “YOUR VNET NAME”


Step 3. (optional) Use Reserved IP for the Azure Cloud Service

To avoid the difficulty of changing IP address when redeploying your Cloud Service, you can reserve a public IP address and assign it when creating a cloud service. This IP address persists even when the cloud service that it is assigned to is deleted.

Create a Reserved IP address (RIP).

  1. NewAzureReservedIP ReservedIPName “RIP NAME” Label “NG Firewall IP” Location “YOUR LOCATION”

Step 4. Create Storage Account

  1. Create a Storage Account and set it as the default storage account.
    1. NewAzureStorageAccount StorageAccountName “STORAGEACCOUNT NAME” Location “YOUR LOCATION”
  2. Use the storage account as the default storage account for this Azure subscription.
    1. SetAzureSubscription SubscriptionName “YOUR AZURE SUBSCRIPTION NAME” CurrentStorageAccountName “STORAGEACCOUNT NAME”
  3. Verify that you are using the correct storage account:
    1. GetAzureSubscription


Step 5. Barracuda NG Firewall Image

You can either create your own image from a VHD file you have uploaded to the storage account, or use the Barracuda NG Firewall image from the Azure Marketplace.

Get ID of the Barracuda NG Firewall image from the Azure Marketplace (Recommended)

To deploy the VM image of the Barracuda NG Firewall from the Azure Gallery, you need to find the exact image name. E.g., for 5.4.3 the image name is: 810d5f35ce8748c686feabed1344911c__BarracudaNGFirewall-5.4.3-182-pl4. The Azure image name changes every time the image in the Azure Gallery is updated.

  1. Open an Azure PowerShell.
  2. Get a list of all available Azure images in the Azure Gallery and only show the ones for the Barracuda NG Firewall and store the image name in a variable. E.g., $image
    1. $image = GetAzureVMImage | where { $_.ImageFamily Match “Barracuda NG Firewall*”} | sort PublishedDate Descending | select ExpandProperty ImageName First 1
Upload a VHD Disk Image and Create the Virtual Machine (Alternative)

If you want to deploy a version of the Barracuda NG Firewall that is not available in the Azure Marketplace, or want to be certain to always deploy the exact same firmware version of the Barracuda NG Firewall, upload a VHD disk image and create your own Virtual Machine.

  1. Download the VHD file from https://login.barracudanetworks.com[5].
  2. Create a new Azure Storage Container.
    New-AzureStorageContainer -Name <name of storage container>

    1. NewAzureStorageContainer Name “images”
  3. Upload the VHD to the Azure storage account.
    Add-AzureVhd -Destination <storage account URL>/<storage container name>/filename.vhd -LocalFilePath -NumberOfUploaderThreads 4

    1. AddAzureVhd Destination https://docstorage02.blob.core.windows.net/images/GWAY-6.0.0-190.vhd -LocalFilePath c:\Azure\GWAY-6.0.0-190.vhd -NumberOfUploaderThreads 4

    Depending on your connection, uploading the disk image might take a long time.

  4. Create a Virtual Machine from the VHD disk image and save the Virtual Machine in a variable so it can be used as a parameter later:
    1. $vmimage = AddAzureVMImage ImageName IMAGE_NAME MediaLocation STORAGE_ACCOUNT_URL/CONTAINER/VHD_DISK_IMAGE_FILE.vhd Label YOUR_LABEL OS “Linux”
    2. $image = $vmimage.ImageName

Step 6. Create and Provision the Azure Configuration for the new Barracuda NG Firewall Virtual Machine

Create the configuration for the new Azure Virtual Machine by defining VM size, the VM image created in step 4, and the Availability Set. If you want to use multiple Network interfaces, you must use a Large or  Extra Large Instance. LargeInstances support two Network Interfaces, Extra Large Instances four Network Interfaces. The LinuxUser parameter is ignored, and the password set is used for the root user on the Barracuda NG Firewall.

  1. $vm1 = NewAzureVMConfig Name “VMNAME” InstanceSize $instanceSize Image $image AvailabilitySetName “NGHACluster”
  2. AddAzureProvisioningConfig Linux LinuxUser “azureuser” Password “SUPERSECRETPASSWORD” VM $vm1 NoSSHEndpoint

Step 7. (optional) Add Endpoints

Add Endpoints for SSH, NG Admin, and all services (e.g., VPN, SSL VPN,..) running on the Barracuda NG Firewall. You can also add Endpoints later.

  1. AddAzureEndpoint Protocol tcp LocalPort 22 PublicPort 22 Name “SSH” VM $vm1
  2. AddAzureEndpoint Protocol tcp LocalPort 807 PublicPort 807 Name “MGMT” VM $vm1
  3. AddAzureEndpoint Protocol tcp LocalPort 691 PublicPort 691 Name “TINAVPN” VM $vm1

Step 8. Assign the Subnet and a Static IP Address to the First Network Interface

You need to assign the subnet in the VNET to the first Network Interface of the Azure Instance. Note that you can define Endpoints only for the first Network Interface of a VM.

  1. Before you assign a static IP address to the VM, check to see if the IP address is available or already in use by another VM
    1. TestAzureStaticVNetIP VNetName “VNET NAME” IPAddress “FRONTEND STATIC IP”
  2. Assign the subnet and
    1. SetAzureSubnet SubnetName “FRONTEND SUBNET NAME” VM $vm1
    2. SetAzureStaticVNetIP IPAddress “FRONTEND STATIC IP” VM $vm1

Step 9. (optional) Add Additional Network Interfaces

Depending on the Azure Instance size, add one or two additional Network Interfaces to your VM. Each Network Interface is assigned a static IP address in their Subnet. You can only use one Network Interface per Subnet.

Limitations of Multiple Network Interfaces in the Azure Public Cloud

  • Multiple NIC is supported on Large and Extra Large Azure VMs. VMs must be in a location-based Azure Virtual Network.
  • Adding or removing NICs after a VM is created is not possible.
  • NICs in Azure VMs cannot act as Layer 3 gateways.
  • Internet-facing VIP RIP is only supported on the first default NIC, and there is only one VIP mapped to the IP of the default NIC. The additional NICs cannot be used in a Load Balance set.
  • The order of the NICs inside the VM will be random, but the IP addresses and the corresponding MACs will remain the same.
  • You cannot apply Network Security Groups or Forced Tunneling to the non-default NICs.
  1. Check if the desired IP address is available:
    1. TestAzureStaticVNET VNetName “VNET NAME” IPAddress “BACKEND STATIC IP”
  2. Add a second Network Interface:
    1. AddAzureNetworkInterfaceConfig Name “NIC2” SubnetName “BACKEND SUBNET NAME” StaticVNetIPAddress “BACKEND STATIC IP” VM $vm1
  3. If you are using an Extra Large Instance, you can add two additional Network Interfaces (four total).

Step 10. Create the Barracuda NG Virtual Machine

You can now create the Barracuda NG Firewall virtual machine.

With a Reserved IP Address:
  1. NewAzureVM ServiceName “CLOUD SERVICE NAME” VM $vm1 ReservedIPName “RESERVED IP NAME” Location “YOUR LOCATION” VNetName “VNET NAME”
Without a Reserved IP Address:
  1. NewAzureVM ServiceName “CLOUD SERVICE NAME” VM $vm1 Location “North Europe” VNetName “VNET NAME”

Step 11. Configure Barracuda NG Admin

You must use the latest version of Barracuda NG Admin to connect to your Barracuda NG Firewall Azure.

You must use Single Point of Entry (SPoE) to connect to the Barracuda NG Firewall in the Azure cloud. SPoE is enabled per default.

  1. Launch NG Admin.
  2. In the upper left-hand corner, click Option and Settings.
  3. Select the check box for SPoE as default.

Next Steps