Discussion:
[rsnapshot-discuss] cloud based backup
Tim Coote
2016-09-06 08:57:55 UTC
Permalink
Hullo

is there any work/success in using rsnapshot with cloud based storage? I’ve noticed that I’m chewing through quite a lot of cash using owned storage and would prefer an approach that used, say, Google Nearline, or AWS glacier storage. However, neither of these supports hard links, so the size of the backups would be huge.

tc
------------------------------------------------------------------------------
Ahmed Kamal
2016-09-06 11:05:36 UTC
Permalink
Checkout http://rclone.org/


On Tue, Sep 6, 2016 at 12:58 PM, Christopher Barry <
On Tue, 6 Sep 2016 09:57:55 +0100
Post by Tim Coote
Hullo
is there any work/success in using rsnapshot with cloud based storage?
I’ve noticed that I’m chewing through quite a lot of cash using owned
storage and would prefer an approach that used, say, Google Nearline,
or AWS glacier storage. However, neither of these supports hard links,
so the size of the backups would be huge.
tc
-----------------------------------------------------------
-------------------
Post by Tim Coote
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
...as would be the time spent pushing them there, and the bandwidth
consumed, and the privacy lost, and the security compromised, and the...
--
Regards,
Christopher
------------------------------------------------------------
------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Patrick O'Callaghan
2016-09-06 12:26:22 UTC
Permalink
Post by Ahmed Kamal
Checkout http://rclone.org/
How does this solve the hard links Issue? It seems to be a file and
directory sync tool, similar to insync (https://www.insynchq.com/) but
supporting more storage providers.

poc

Tim Coote
2016-09-08 10:24:25 UTC
Permalink
We're seeing what I think is reasonable failure rates: 6tb reds are between
2 and 2.5% per year. The 8's are slightly higher, but we're still running
numbers on those. Overall the "slowness" of the volume is the drives, not
gluster or zfs. We toss a 100gb-ish ZIL drive in the box as well, so that
helps, and run OS on a single drive. If it fails and a box drops out, oh
well, there's plenty of redundancy built in. Re-syncs can take a while,
depending on how long that node was "missing", but we havne't had two
completely fail at the same time, yet.... Speaking of, we're running
infiniband interconnects between all the nodes, so that part is fast. I
didn't include the cost of that in the $70K figure though, but 10gbe works
as well and is cheap.
Anyway, in the interests of keeping things simple, this is working pretty
well for us. Turning off atime helps, as does the ZIL. And not running
dedup, of course, but we know our data will never, ever, be duplicated.
sudo zfs create -o atime=off -o compression=lz4 -o exec=off -o xattr=sa -o
acltype=posixacl tank/gluster
I've got a basic example howto mostly written up that I can send to anybody
interested, minus some of the site specific stuff we do.
https://www.supermicro.com/products/system/4u/6048/ssg-6048r-e1cr24h.cfm
(.......and, if you order right now, it'll cost less than $70K/pb. Call
1-800-netapp-sux, that's 1-800-netapp-sux, 800-netapp-sux!)
kw
That sounds like a reasonable price. But what’s the price to the business of the service? The overall costs would include utilisation, DC, staff and financing costs + other overheads, which tend to dominate the costs of using any technology.

For comparison, the Glacier equivalent, with a recovery time of a few hours - so it may not be appropriate - is 252k to 336k for 3 or 4 year use.
------------------------------------------------------------------------------
Ken Woods
2016-09-06 15:28:40 UTC
Permalink
Just as a curiosity, what's your base data size, and how much change do you have per month?

We are standing up zfs under gluster for less than $70k/PB
It can be done for $60k if you stuff drive into the middle of a chassis.
It'd really make my day if you're using netapps. Please tell me you are.
Post by Tim Coote
Hullo
is there any work/success in using rsnapshot with cloud based storage? I’ve noticed that I’m chewing through quite a lot of cash using owned storage and would prefer an approach that used, say, Google Nearline, or AWS glacier storage. However, neither of these supports hard links, so the size of the backups would be huge.
tc
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
------------------------------------------------------------------------------
Ken Woods
2016-09-06 22:44:55 UTC
Permalink
We're seeing what I think is reasonable failure rates: 6tb reds are between
2 and 2.5% per year. The 8's are slightly higher, but we're still running
numbers on those. Overall the "slowness" of the volume is the drives, not
gluster or zfs. We toss a 100gb-ish ZIL drive in the box as well, so that
helps, and run OS on a single drive. If it fails and a box drops out, oh
well, there's plenty of redundancy built in. Re-syncs can take a while,
depending on how long that node was "missing", but we havne't had two
completely fail at the same time, yet.... Speaking of, we're running
infiniband interconnects between all the nodes, so that part is fast. I
didn't include the cost of that in the $70K figure though, but 10gbe works
as well and is cheap.

Anyway, in the interests of keeping things simple, this is working pretty
well for us. Turning off atime helps, as does the ZIL. And not running
dedup, of course, but we know our data will never, ever, be duplicated.

sudo zfs create -o atime=off -o compression=lz4 -o exec=off -o xattr=sa -o
acltype=posixacl tank/gluster

I've got a basic example howto mostly written up that I can send to anybody
interested, minus some of the site specific stuff we do.

And we're using these, at least this year:
https://www.supermicro.com/products/system/4u/6048/ssg-6048r-e1cr24h.cfm

(.......and, if you order right now, it'll cost less than $70K/pb. Call
1-800-netapp-sux, that's 1-800-netapp-sux, 800-netapp-sux!)

kw
Interesting .. how are you finding gluster stability and any gotchas with
zfs underneath it ?
To the OP, if you can switch to ZFS, it might be your best option
Post by Ken Woods
Just as a curiosity, what's your base data size, and how much change do
you have per month?
We are standing up zfs under gluster for less than $70k/PB
It can be done for $60k if you stuff drive into the middle of a chassis.
It'd really make my day if you're using netapps. Please tell me you are.
Post by Tim Coote
Hullo
is there any work/success in using rsnapshot with cloud based storage?
I’ve noticed that I’m chewing through quite a lot of cash using owned
storage and would prefer an approach that used, say, Google Nearline, or
AWS glacier storage. However, neither of these supports hard links, so the
size of the backups would be huge.
Post by Tim Coote
tc
------------------------------------------------------------
------------------
Post by Tim Coote
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
------------------------------------------------------------
------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Tim Coote
2016-09-08 10:26:44 UTC
Permalink
Date: Thu, 8 Sep 2016 11:05:47 +0100
Subject: Re: [rsnapshot-discuss] rsnapshot-discuss Digest, Vol 123,
Issue 6
Replying to a list digest with a digest of replies? I've never seen that
before.
If you really need to get the list via a digest (hint: you almost certainly
don't as it's an ancient kludge designed for people with modems and UUCP),
then please reply to each message separately, otherwise it will quickly
become impossible to handle. Most modern mail clients can do this in such a
way as to preserve message threading, which means among other things not
uselessly showing the name of the digest as the Subject line.
Cheers
poc
I know. I was being lazy as the whole digest was my original post + follow-ups.
Sorry.
------------------------------------------------------------------------------
Continue reading on narkive:
Loading...