If you have created a new VD using a disk or group of disks that were previously partitioned, you may see a message like this when you try to initialize the virtual disk for LVM.

# pvcreate /dev/sdd
  Device /dev/sdd excluded by a filter.

If filter is correct (/etc/lvm/lvm.conf) and old partition table information found on the disk, you can wipe out the old partition information with “wipefs”.

# wipefs -a /dev/sdd
/dev/sdd: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
/dev/sdd: calling ioclt to re-read partition table: Success

Now, try pvcreate command again.

# pvcreate /dev/sdd
Physical volume "/dev/sdd" successfully created.

Cockpit packages

Package NamePurpose
cockpit-dockerManaging Docker Containers
cockpit-kubernetesVisualizing and Configuring Kubernetes Cluster
cockpit-machinesManage KVM Virtual Machines
cockpit-sosreportCreate diagnostic report with the sosreport tool
cockpit-selinuxTroubleshoot SELinux Issues
cockpit-kdumpConfigure Kernel Crash Dumps
cockpit-subscriptionsManage System subscription
cockpit-machines-ovirtManage oVirt Virtual Machines
cockpit-pcpReading PCP metrics and Loading PCP archives

Disk > Mdadm > LVM


mdadm -D /dev/md127

watch cat /proc/mdstat

# Create vg1

vgcreate vg1 /dev/md127

# Create lg disk

lvcreate -L 150M vg1 -n lv1
lvcreate -L 150M vg1 -n lv2
lvcreate -L 150M vg1 -n lv3



mkfs.ext4 /dev/vg1/lv1

mkfs.ext4 /dev/vg1/lv2

mkfs.ext4 /dev/vg1/lv3

mkdir /home/lv1 && mount /dev/vg1/lv1 /home/lv1

mkdir /home/lv2 && mount /dev/vg1/lv2 /home/lv2

mkdir /home/lv3 && mount /dev/vg1/lv1 /home/lv3

#Extend lv disk, first check for free PE of PV disk with pvdisplay

lvextend -l +200 /dev/vg1/lv1

resize2fs /dev/vg1/lv1

lvextend -l +100 /dev/vg1/lv4

xfs_growfs /dev/vg1/lv4

lvextend -L50G -n /dev/vg1/lv4

xfs_growfs /media/root/87257771-0931-4d1c-b538-58e9c6b1e09b

###Note: fill all space of PV disk: lvextend -l +100%FREE /dev/vg1/lv4



How to Backup AWS S3 Buckets

The S3 Backup Challenge

When migrating a large file system to AWS S3, One of the surprising challenges is the implementation of a backup strategy.  With file-system based storage, you often back up the disk with periodic snapshots (daily, weekly, monthly, etc).   There is comfort in knowing that you have a plethora of time capsuled copies stored away and, in the event of an extinction level event, can recover most of your data.  
It turns out this analogy doesn’t easily transfer over to S3.  In fact, after investigating the confusing alternatives available for S3 backup, it can feel like you are ending up with less recoverability than you have now.  The reality is that you need to think about backup a little differently and focus on the recovery scenarios.    In the final analysis, you are probably better off with S3 in this light than with traditional disk snapshots.  

S3 Recovery Scenarios

Assuming that you have researched S3, you probably want to handle the following scenarios – 

  1. S3 bucket fails.  Unlikely but the equivalent of a disk crash.  
  2. Accidentally delete objects.  Human-error on our side.  
  3. Permanently delete object versions.  Again, human error on our side.
  4. Objects/data get corrupted.   Backups could propagate to multiple versions
  5. Accidentally delete bucket.   The apocalypse scenario.  The equivalent of rm -rf mybucket/

1. S3 Bucket Fails – Solution: Cross Region Replication

Problem: AWS has an infrastructure failure in S3 or the surrounding data center.  AWS claims > 99.99% reliability in S3 so this scenario is unlikely however the solution is to have the entire bucket copied to another bucket somewhere else.  Fortunately, AWS provides a built-in mechanism to achieve this – Cross Region Replication (CRR).

Steps to enable CRR

1.  Create a new bucket to replicate data into.  Eg. mybucket-replica.  Create it in a different region.  Eg.  If your original bucket was in us-east1, put the replication bucket in us-east2.  Make sure to enable versioning on the bucket.2.  On the original bucket, go to Management->Replication and Add Rule.  Again, versioning must be enabled on your original bucket.3.  [Source] To replicate all content (going forward), keep the defaults and hit Next.  [Destination] Select the new bucket that you created (Eg. mybucket-replica).  Click Next.  [Permissions] Select Create new role.  Review and click Save.

Note:  Replication does not retroactively copy objects that were already in the bucket so you may need to manually copy over existing objects.

2.  Accidentally Delete S3 Objects –  Solution: S3 Bucket Versioning

This scenario covers perhaps the most common case which is your own code or process causes objects to get deleted inadvertently.  Fortunately, S3 versioning handles this case automatically; and better than “brick-level” restore from snapshots.  With versioning enabled, objects are not actually deleted from buckets.  While you will reference the “current” version, you can revert to previous versions at any time.  We won’t go through the mechanics of restoring previous versions here but it is important to enable versioning on the bucker.

Steps to enable S3 bucket Versioning

1.  Go into the bucket properties.  Click on Versioning and then Enable versioning.

3.  Permanently Delete Object Versions –  Solution: MFA Delete Protection (versioning)

So would if you accidentally delete old (all) previous versions of an object?  To prevent this from occurring, you can require Multi-Factor Authentication (MFA) for all permanent deletions of objects.  So an administrator will have to provide a code from an MFA-enabled device to achieve this deletion making it very unlikely to happen without the deliberate intent to delete that object.
There is a surprising lack of detail surrounding both the conceptual approach and the mechanics of implementing this.   Part of the confusion is that there are two different contexts for “MFA Delete”.  In this case we are talking about an MFA Delete property on the bucket’s versioning property.  This cannot be set from the console.  You have to use the AWS CLI to enable this but before doing that you need MFA set up on the account and an MFA device.  Think of an MFA device as being one of the RSA SecurId fobs with a constantly changing 6 digit number.

Steps to enable S3 MFA Delete on a bucket

1.  First, make sure that you have MFA enabled on your account (root).  IAM -> Activate MFA on your root account.  You will need to set up a device.  The easiest option is to download the Authenticator app from Google onto your phone.  You will then scan the QR code shown in the AWS console on your phone and you have now have MFA enabled.
2.  Now you can set up MFA Delete on the versioning information for the bucket.  Again, to do this you need to use the CLI.  Here is a command to set this up.  Note that the ‘123456’ that you see below is an actual code shown on your MFA device (Authenticator app on your phone).aws s3api put-bucket-versioning –bucket mybucket –versioning-configuration Status=Enabled,MFADelete=Enabled –mfa “arn:aws:iam::111111222222:mfa/root-account-mfa-device 123456”
3.  While you cannot set this property in the console, you can see it.  If you click on the bucket, you should see the properties on the right where you can verify that MFA delete is enabled.

4.  Objects/data get corrupted –  Solution: S3 Bucket Versioning

Another scenario that can occur is that some portion of your data gets corrupted.  This corruption, if left untreated, can propagate to multiple snapshots or versions.  With a finite number of backups, this could result in ALL of the snapshots becoming corrupted.  Fortunately, if you don’t expire out old versions, your S3 bucket can contain a full version history of each object making recovery as straightforward as finding the last uncorrupted version.

Steps to enable S3 MFA Delete on a bucket

(See scenario 2 above to enable bucket versioning)

5.  Accidentally Delete Bucket –  Solution: MFA Delete Bucket Policy

Finally, we have the scenario that is most unsettling; the entire bucket getting inadvertently deleted. This feels much worse than any scenario is in the disk/snapshot world because this is like losing the current file system AND ALL BACKUPS at the same time.
First of all remember that we enabled cross-region replication in step 1. So, we do have another copy of the entire bucket with ALL versions somewhere else AND it is up to date.
The key here is to make it virtually impossible to delete the bucket.  We do this by setting a bucket policy that denies delete attempts without MFA; similar to the MFA delete on the bucket versioning but set up differently.  For this, we use a bucket policy. 
Now, while the bucket policy that we put in place will require MFA, remember that a user can log into the console using MFA.  Technically, they now have an MFA authentication and could delete the bucket from the console.  A nice secondary check that you can include in a policy is the age of the authentication; that is, how long ago did the user authenticate.  One precaution that you can take is to set this time, the MultiFactorAuthAge, to 1 second!  This will make it impossible to achieve from a log into the console.  So now the only way to achieve the delete is to first change the policy.

Steps to enable S3 MFA Delete on a bucket

1.  Go into the bucket and click Permissions then Bucket Policy2.  Enter or add something similar to the following policy

    "Version": "2012-10-17",
    "Statement": [
            "Sid": "MFADeleteOnly",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:DeleteBucket",
            "Resource": "arn:aws:s3:::myBucket",
            "Condition": {
                "NumericGreaterThanIfExists": {
                    "aws:MultiFactorAuthAge": "1"


With a little thought you can create a highly available S3-based storage environment that is resilient enough to recover from virtually all of the common mayhem scenarios.  While the approach differs from traditional file system snapshots (backups), in many ways it provides better recovery in practice.

Install minio Ubuntu

Minio is a self-hosted solution for creating your own Object storage. It is an alternative for AWS S3, if you have used that service before. The Minio software itself is shipped as a plain binary and even the official documentation suggests you to use it that way, instead of using a package manager. There are, of course, Docker images if you wish to use those to run minio on your VPS.

In this tutorial we will be installing and demonstrating the use of Minio on Ubuntu 18.04 LTS server. This VPS has a static IP and I will be setting up DNS records and TLS connections to make this Object store as secure and production ready as possible.


Here are the prerequisites you would need if you want to follow along:

  1. A VPS running Ubuntu or any other Linux distro with a static IP (IP_ADDRESS will be our placeholder, replace it with your VPS’ actual IP address)
  2. A Fully Qualified Domain Name [FQDN]. will be our placeholder.

Installation and Miscellaneous Setup

Let’s login to our VPS and get things ready for Minio to run properly.

1.  DNS Setup

Go to the nameserver where your domain’s DNS records are maintained, most likely this is found at your domain registrar’s website. Add an A record, pointing your chosen FQDN (for example ) to your VPS’ IP_ADDRESS.

2.  Minio User

Before we install Minio, let’s create a new UNIX user account under whom minio will run. We don’t want to run it as root or as the regular user who may have sudo access or other applications running under it. We create a minio system account, named minio-user:sudo useradd –system minio-user –shell /sbin/nologin

3.  Minio Download

Next we download the minio binary (It is written in Go which compiles into a small lightweight binary).

Get the binarycurl -O

Move the binary to a location where binaries are usually expected to reside:sudo mv minio /usr/local/bin

Make the binary file executable and give minio-user user and group its ownership:$ sudo chmod +x /usr/local/bin/minio
$ sudo chown minio-user:minio-user /usr/local/bin/minio

4.  /etc config files, startup scripts and storage device

We need Minio to start up with system reboot and be recognized as a running service by the OS. Not doing so would result in catastrophes such as when the OOM-killer sees this process and decides that it’s not useful enough. We would also need a directory where the actual data of our object store is going to be saved:$ sudo mkdir /usr/local/share/minio
$ sudo mkdir /etc/minio

Make sure that minio has full control over these directories:$ sudo chown minio-user:minio-user /usr/local/share/minio
$ sudo chown minio-user:minio-user /etc/minio

Inside the /etc/default directory we need to create a minio file to specify environment variables like the port number we will be listening on and the directory where the data  should be saved (the volume). We created the volume earlier that was the /usr/local/share/minio directory. So use your favourite text editor to create a file /etc/default/minio and add the following contents inside it:

MINIO_OPTS=”-C /etc/minio –address″

Make sure that you write your VPS’ actual designated FDQN instead of the literal string above. Port number 9000 is what they usually use in the Documentation but we are going to use proper TLS installation listening at port 443. Since it is a port number less than 1024, we need to expressly tell the OS that it is okay for minio to listen on these ports:sudo setcap ‘cap_net_bind_service=+ep’ /usr/local/bin/minio

Lastly, we need to configure the minio service. Fortunately the script that does it is available at their GitHub repo and we will place it at its appropriate place:$ curl -O
$ sudo mv minio.service /etc/systemd/system

Reload all systemd units and enable minio to start on boot$ sudo systemctl daemon-reload
$ sudo systemctl enable minio

Lastly, make sure that your firewall allows communication at port 443.

LetsEncrypt TLS Certificates using Certbot

We need to negotiated TLS certificates between our Minio server and LetsEncrypt. Certbot is the client which does this for us and also automates the certificate renewals. Let’s first install Certbot:$ sudo apt update
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot

Then install the certs, as documented by Minio Docs:$ sudo certbot certonly –standalone -d –staple-ocsp -m –agree-tos

Here you enter your FQDN for the Minio server after the -d flag and your email address after -m flag. The email address is important as it allows LetsEncrypt to notify you about pending renewals.

Your emails will now be present at /etc/letsencrypt/live/ Of course, the last directory name would depend on your chosen FQDN. Now copy the certs to Minio’s /etc/minio directory and give it permission to access them.

$ cp /etc/letsencrypt/live/ /etc/minio/certs/public.crt
$ cp /etc/letsencrypt/live/ /etc/minio/certs/private.key
$ chown minio-user:minio-user /etc/minio/certs/public.crt
$ chown minio-user:minio-user /etc/minio/certs/private.key

Now you are ready to use the service:$ sudo service minio start
$ sudo service minio status


  • minio.service – Minio

Loaded: loaded (/etc/systemd/system/minio.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2018-10-09 11:54:41 PDT; 5s ago
Process: 15874 ExecStartPre=/bin/bash -c [ -n “${MINIO_VOLUMES}” ] && echo “Variable
MINIO_VOLUMES not set in /etc/default/minio” (code=exited, status=0/SUCCESS)
Main PID: 15877 (minio)
Tasks: 13 (limit: 4915)
CGroup: /system.slice/minio.service
└─15877 /usr/local/bin/minio server -C /etc/minio –address /usr/

Oct 09 11:54:41 hostname minio[15877]: Browser Access:
Oct 09 11:54:41 hostname minio[15877]:
Oct 09 11:54:41 hostname minio[15877]: Command-line Access:
Oct 09 11:54:41 hostname minio[15877]: $ mc config host add myminio
PAMH22LU3YJIFLU82H2E IKJ+qtc0Oqdn46R3dLfsjv5bCnm8UEeIBYpdG8qg

The output of this command will contain the access key (PAMH22LU3YJIFLU82H2E) and secret key (IKJ+qtc0Oqdn46R3dLfsjv5bCnm8UEeIBYpdG8qg) for minio as shown in bold letters above. Your keys would be different so don’t copy the ones mentioned here.

Using Minio

Open up your browser and visit (make sure to use the FQDN you assigned) and use the access and secret key listed in your service minio status command to login in for the first time.

restic with MinIO Server

restic is a fast, efficient and secure backup tool. It is an open source project available under BSD 2-Clause License.

In this recipe we will learn how to use restic to backup data into MinIO Server.

1. Prerequisites

Install MinIO Server from here.

2. Installation

Install restic from

3. Configuration

Set MinIO credentials in the environment variables as shown below.


4. Commands

Start restic and point it to the bucket where the backup data will reside.

Copy./restic -r s3:http://localhost:9000/resticbucket init

Copy backups from the local machine to the bucket on MinIO server.

Copy./restic -r s3:http://localhost:9000/resticbucket backup /home/minio/workdir/Docs/
enter password for repository:
scan [/home/minio/workdir/Docs]
scanned 2 directories, 6 files in 0:00
[0:00] 100.00%  0B/s  8.045 KiB / 8.045 KiB  6 / 8 items  0 errors  ETA 0:00
duration: 0:00, 0.06MiB/s
snapshot 85a9731a saved