spec: implement compatibility with segwit.

[chjj]

This is something I've considered for a while. I've been thinking about it more frequently for the past week or two: mainchain segwit adds a few benefits to extblk. The exclusion of mainchain segwit wasn't a very conscious technical decision initially. It was just the simplest path for the code at the time, and less complexity. I think this decision should be reexamined technically for this reason.

I've revised some major parts of the specification, mostly pertaining to how entrance into and exits from the extension block behave. The new behavior is similar to Johnson's 2017 proposal. Listing changes and potential benefits here.

Changes

Consensus

• The extblk deployment now depends on the segwit deployment. This is a side effect of using witness programs as entrances/exits (both deployments need to be able to coexist in this setup).
• The commitment magic number has been changed from 0xaa21a9ef to 0x45424c4b (ascii: EBLK). This is to avoid confusion.
• The commitment itself no longer includes canonical block txids, since this is already included by the segwit commitment.
• Entrances into the extension block must be witness programs v9 through v16. These are reserved ahead of time so the protocol knows which outputs need to enter/exit. The first defined witness programs will be v9, replicating the current v0 programs.
• Exits from the extension block must only be witness programs v0-v8. This is probably the biggest benefit of including mainchain segwit, and mitigates potential edge cases with legacy wallets trying to spend early from a resolution output (segwit-enabled wallets should be aware of the new maturity rule). See Exit maturity #9. cc @jl2012

P2P

• The extension block payload is now always delivered when a getdata inv type is used with the witness bit.
• Extension transactions are now always delivered with the getdata witness bit.
• The witness service bit could potentially be used to signal both extblk+segwit support, since (I think) trailing bytes on a block message are simply ignored by a lot of implementations (someone please correct me here if I'm wrong). Backward compatibility is sort of built-in.

Policy

• We can now drop the notion of a mempool nursery for early resolution spends (mentioned elsewhere). The mempool will simply reject early spends of resolution outputs. This ensures segwit wallets won't mistakenly see an early spend of a resolution output coming to them, and in turn also mistakenly spend from that.

Other considerations / Todo

• SIGHASH_NOINPUT (as described in the LN paper) could potentially be implemented in segwit's sighash algorithm. This requires for a modification to mainchain segwit (either in a new wit program or in v0 if the deployment is renewed), but could allow for safe early spends of resolution outputs (with some caveats). It also adds the potential for Lightning outsourcing features -- for this purpose, we could have it in the extension block right off the bat.
• A non-contextual check for which chain the transaction resides in could (should?) be added. I'm thinking of two methods. Either:
  • Transactions within the extension block must have the second segwit flags bit set to 1 (implementations must now track this). This gives an easy non-contextual check for which chain the transaction is in, which has nice implications for wallets and protocols like compact block relay. Note that the flags byte is malleable in the mempool. I also don't think the flag byte was intended to be used this way, but rather for future serialization changes.
  • Transactions within the extension block must have a high version bit set. This has similar implications to the flag bit, without the malleability problem since the version is signed. The issue here is that it makes some parts of code less reusable, since we'll have to and off the low bits to see the "real" version number. This also requires a new rule requiring this bit to never be used on the canonical chain.

I have a preliminary reference implementation here: https://github.com/bcoin-org/bcoin-extension-blocks/tree/extblk-sw2. Everything outlined above is more or less present, but the code still needs some work and more tests.

If consider this route, I think we can have extnet2 up in a reasonable amount of time to play around with this.

Opening it up for discussion. Thoughts?

cc @gasteve @josephpoon

[afk11]

I'm pretty sure LN no longer requires SIGHASH_NOINPUT, but maybe someone else can advise better?

[chjj]

@afk11, segwit gave LN the malleability fix it needed, but LN can still benefit from SIGHASH_NOINPUT for easier outsourcing of blockchain monitoring. A third party can create a penalty transaction on your behalf, assuming you give them a half-created transaction and a signature ahead of time.

Explained here: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-February/012460.html

However, it is very useful for LN if there is a certain level of
outsourcibility for transactions without this 3rd party taking on
onerous costs. In LN, there is a dispute resolution period established
to prevent the counterparty from attesting an incorrect channel state
(represented by broadcasting a timelocked transaction). In other words,
if someone in a channel broadcasts an incorrect state, the output can be
redeemed by a 3rd party (but this 3rd party is not a custodian, since
the output goes to the other party in the channel).

There may be other use cases for LN I'm not aware of. @josephpoon or @Roasbeef would be able to fill in the blanks better than I can.