First CBT questions

[olivierlambert]

Hi lads!

After a first documentation reading, I have a some questions regarding this (by the way, thanks for the doc, it was clear).

• where is the VHD format? I only see RAW everywhere. Sounds like a bit strange since delta VHD already worked fine. VHD is a pretty good format (except 2TiB limit that could be overcome with VHDX), also documented. Is VHD support for CBT is planned?
• doc said "other backup solutions export a snapshot of the whole VDI every single time the VDI is backup up": that's not true, I'm a bit surprised about this statement. We export a delta between two VDI, and XS gives us the delta VHD, which is perfectly fine (since 6.5!). The whole paragraph is talking about existing delta VHD feature that we already use in Xen Orchestra, so I don't get why it said it wasn't possible without CBT.
• if you have a non-thin pro SR used at more than 50%, you still can't backup all VMs at once, CBT or not. In other words, it doesn't change the problem at all (vs our current solution). Is it possible to create directly a CBT snapshot without having to "remove data" then?
• why having an extra dependency (nbd) since VHD delta works? I'm probably missing something, could you explain a bit?

In the end, the only functional advantage I can see against current delta generation, is to remove snapshot data after the snapshot is made. But IMHO it wasn't the major blocker. In the end, non thin pro SR with the base copy mechanism that double the VM disk space issue is the main problem, that CBT can't solve (if it's removing snapshot after). I'm probably wrong somewhere because I just read the doc, but I'll be happy to discuss to see where did I missed it :)

[mauzilla]

Jeez I am late to the party but hello everyone!

I have spent the last week or 2 going through the documentation and have a "MVP" but see Olivier has asked some very interesting questions and would like to weigh some support.

I was amazed to see that the exports are indeed full VDI's for first base and although subsequent backups are "deltas", this leads to a scenario where to rebuild the base (for example if we want to rotate 7 days etc) a new base file needs to be created requiring double the space on the storage. Our current backups (albeit full exports) atleast dont backup a VM at full "size" but only written blocks. We have atleast 4 or 5 VM's that have disks each of 2TB which will now be written as raw data instead of only the real written blocks.

It also seems that having the rebuild a VDI by hand each night isn't really a very attractive solution. Most backups require rotation (we cant keep indefinate incremental backups and would have to rewrite to base sooner or later) which again will require an additional 2TB of storage for each base update.

The XAPI documentation has an alternative solution (which I have not tried but believe is used by Olivier's team). From what I can gather the biggest flaw there is that you cannot delete the contents of the VDI snapshots but on the other hand you sacrifice requiring a massive backup storage pool.

Are there any plans to look at VHD solutions? The only reason we are going with the CBT option so far was simply because it was Citrix supported but as all of our storage repos are thin having backup storage requirements that at full VDI's feels wrong.

I have not even gotten to the point of writing back to the hypervisor yet and hope that the NBD client will allow us to write blocks back to a DR VM.

[olivierlambert]

@mauzilla have you tried Xen Orchestra? It works pretty well, and storage space isn't an issue (on the SR, with thin pro SR).

Note that we recently took a look back at NBD, for one reason only: it's more "generic" and future storage (not using SMAPIv1) won't have VHD format anyway. So it's up to the backup solution to "convert" this NBD raw content to something else (eg qcow2, so you won't have 2TiB limit on backup side).

[mauzilla]

@olivierlambert : I have tried XO, loved it, but we're not quite there yet regarding size and we like your enterprise features so I don't mind seeing how much I can do. If we will ever subscribe to an enterprise solution, we will definitely use XO

I think for those wanting to use the CBT solution, there are definite benefits over traditional XVA exports:

• Hypervisors really have less disk activity
• Can perform faster backups thus make backups more frequently
• Although you have to rebuild the base by hand (took me days to figure out I cannot simply update the base as although you can add new data to the base by python seek, you will add the new block data and it wont overwrite the previous block), I think it may be faster to rebuild than exporting the XVA over network. One has to however add that you are now dealing with VDI's in their full size, which may mean slower rebuilds than smaller XVA exports. Will do some benchmarking

Biggest negative is the VDI exports are not thin (so raw data) which is less than ideal, but don't know if there are any other options.

Hopefully I can get NBD write to work so that we can simply update a failover VDI the same way you use the NBD client to get the block data.

[olivierlambert]

Have you missed the fact that XO is fully Open Source and you can use it without any paid subscription? (but no support and you have to git pull it yourself). See https://xen-orchestra.com/docs/installation.html#from-the-sources

Don't re-invent the wheel 😉

Also XO doesn't use XVA export for delta, it's using directly VHDs.

[mauzilla]

I did not know that at all, will install tomorrow thank you for the tip :)

[mauzilla]

@olivierlambert, I've installed XO and happy to say it works. I have tested a couple of things internally and overall great product, good job, I've been developing these scripts the past couple of years and know the efforts that goes into making a product like yours work. I don't want to use it in our production environment as a primary backup as we want to contribute to your services (and obviously get support if we ever need it)

As I've invested a couple of weeks in this (and years of "I'm going to do this") I want to round off what I have done so far (which I also dont feel is "production" ready) which is 1 simple issue. Although the rebuild issue persists, I do however understand that this is a shortfall on all delta backup solutions, that to restore you need to rebuild, makes sense now. What I did not anticipate was that on restore (using the CBT examples here) I would need to recreate a VDI each time at the restoration SR. As we dont just want to backup but also prepare for failover scenarios, this makes it impossible for large VM's if I need to reimport the entire base each time. For large VM's this would mean a lot of overhead on all drives involved.

So before I embark on the next couple of weeks of troubles, I wonder if its possible to write back to the "failover" servers VDI's using nbd write. That would give in my opinion the best solution and creates an opportunity for delta backups with nbd, a centralised backup location and then the option to reimport that exact nbd export to the target "failover" server. I intend on testing this the next couple of days and will keep all posted, but would appreciate inputs if this is a useless exercise.

[mauzilla]

Infact, why not setup a nbd server on the backup server and set it up against the base of the VDI (at backup storage) then download the changed blocks from source server along with bitmap data. Then instead of rebuilding the base VDI during retention, you could simply write back the "snapshot" using NBD to the base file and for failover purposes connect to the failover server and write changes to the appropriate VDI.

Sounds interesting enough, will share my findings in a couple of days

[olivierlambert]

In theory, you can do that with VHD (directly write VHD content to a target VDI). So it should be possible to "inject" a delta content on top of an active VDI. However, the main issue is to inject a delta on a correct "base" (snapshot). If you mess up, then you corrupt everything.

So it's not a matter of NBD, it's a matter of doing something that's logical for your data.

Also, I don't think that's really needed: if you want to rollback, just use snapshots (rolling snapshot feature of XO). And if you just want to restore a file, use file level restore of XO.