change db schema, allow TXO-spender lookups, protocol 1.5

docs/protocol-basics.rst

@@ -68,14 +68,12 @@ revision number.
 
 A party to a connection will speak all protocol versions in a range,
 say from `protocol_min` to `protocol_max`, which may be the same.
-When a connection is made, both client and server must initially
-assume the protocol to use is their own `protocol_min`.
 
-The client should send a :func:`server.version` RPC call as early as
-possible in order to negotiate the precise protocol version; see its
-description for more detail.  All responses received in the stream
-from and including the server's response to this call will use its
-negotiated protocol version.
+The client must send a :func:`server.version` RPC call as the first
+message on the wire, in order to negotiate the precise protocol
+version; see its description for more detail.
+All responses received in the stream from and including the server's
+response to this call will use its negotiated protocol version.
 
 
 .. _script hashes:
@@ -155,10 +153,11 @@ block)
 
   * ``height`` is the height of the block it is in.
 
-3. Next, with mempool transactions in any order, append a similar
-string for those transactions, but where **height** is ``-1`` if the
-transaction has at least one unconfirmed input, and ``0`` if all
-inputs are confirmed.
+3. Next, with mempool transactions ordered
+:ref:`as described here <mempool_tx_order>`,
+append a similar string for those transactions,
+but where **height** is ``-1`` if the transaction has at least
+one unconfirmed input, and ``0`` if all inputs are confirmed.
 
 4. The :dfn:`status` of the script hash is the :func:`sha256` hash of the
 full string expressed as a hexadecimal string, or :const:`null` if the

docs/protocol-changes.rst

@@ -2,7 +2,7 @@
 Protocol Changes
 ================
 
-This documents lists changes made by protocol version.
+This document lists changes made by protocol version.
 
 Version 1.0
 ===========
@@ -159,10 +159,36 @@ Changes
     Non-AuxPoW chains are unaffected.
 
 
-Version 1.4.1
+Version 1.4.2
 =============
 
 New methods
 -----------
 
-  * :func:`blockchain.scipthash.unsubscribe` to unsubscribe from a script hash.
+  * :func:`blockchain.scripthash.unsubscribe` to unsubscribe from a script hash.
+
+.. _version 1.5:
+
+Version 1.5
+===========
+
+Changes
+-------
+
+  * Breaking change for the version negotiation: we now mandate that
+    the :func:`server.version` message must be the first message sent.
+    That is, version negotiation must happen before any other messages.
+  * The status of a scripthash, :func:`blockchain.scripthash.get_mempool`,
+    and :func:`blockchain.scripthash.get_history` previously did not define
+    an order for mempool transactions. We now mandate that these are sorted
+    by `(-height, tx_hash)`.
+  * The previously required *height* argument for
+    :func:`blockchain.transaction.get_merkle` is now optional.
+  * Optional *mode* argument added to :func:`blockchain.estimatefee`.
+
+New methods
+-----------
+
+  * :func:`blockchain.outpoint.subscribe` to subscribe to a transaction
+    outpoint, and get a notification when it gets spent.
+  * :func:`blockchain.outpoint.unsubscribe` to unsubscribe from a TXO.

docs/protocol-methods.rst

@@ -160,12 +160,19 @@ be confirmed within a certain number of blocks.
 
 **Signature**
 
-  .. function:: blockchain.estimatefee(number)
+  .. function:: blockchain.estimatefee(number, mode=None)
+  .. versionchanged:: 1.5
+     *mode* argument added
 
   *number*
 
     The number of blocks to target for confirmation.
 
+  *mode*
+
+    A string to pass to the bitcoind *estimatesmartfee* RPC as the
+    *estimate_mode* parameter. Optional.
+
 **Result**
 
   The estimated transaction fee in coin units per kilobyte, as a
@@ -362,8 +369,9 @@ hashes>`.
 
 **Result**
 
-  A list of mempool transactions in arbitrary order.  Each mempool
-  transaction is a dictionary with the following keys:
+  A list of mempool transactions, ordered by `(-height, tx_hash)`
+  (see :ref:`below <mempool_tx_order>`).
+  Each mempool transaction is a dictionary with the following keys:
 
   * *height*
 
@@ -377,6 +385,14 @@ hashes>`.
 
     The transaction fee in minimum coin units (satoshis).
 
+
+  .. _mempool_tx_order:
+
+  Mempool txs are ordered by `(-height, tx_hash)`, that is,
+  ``0`` height txs come before ``-1`` height txs, and secondarily the
+  txid (with endianness same as displayed hex) is used to arrive at
+  a canonical ordering.
+
 **Result Example**
 
 ::
@@ -498,6 +514,109 @@ Unsubscribe from a script hash, preventing future notifications if its :ref:`sta
   Note that :const:`False` might be returned even for something subscribed to earlier,
   because the server can drop subscriptions in rare circumstances.
 
+blockchain.outpoint.subscribe
+===============================
+
+Subscribe to a transaction outpoint (TXO), to get notifications about its status.
+A status involves up to two transactions: the funding transaction that creates
+the TXO (as one of its outputs), and the spending transaction that uses it
+as an input (spends it).
+
+**Signature**
+
+  .. function:: blockchain.outpoint.subscribe(tx_hash, txout_idx)
+  .. versionadded:: 1.5
+
+  *tx_hash*
+
+    The TXID of the funding transaction as a hexadecimal string.
+    (sometimes called prevout_hash, in inputs)
+
+  *txout_idx*
+
+    The output index, a non-negative integer. (sometimes called prevout_n, in inputs)
+
+**Result**
+
+  The status of the TXO, taking the mempool into consideration.
+  The output is a dictionary, containing 0, 1, or 3 of the following items:
+
+  * *height*
+
+    The integer height of the block the funding transaction was confirmed in.
+    ``0`` if the funding transaction is in the mempool.
+    This key must be present if and only if there exists a funding transaction
+    (either in the best chain or in the mempool), regardless of spentness.
+
+  * *spender_txhash*
+
+    The TXID of the spending transaction as a hexadecimal string.
+    This key is present if and only if there exists a spending transaction
+    (either in the best chain or in the mempool).
+
+  * *spender_height*
+
+    The integer height of the block the spending transaction was confirmed in.
+    ``0`` if the spending transaction is in the mempool.
+    This key is present if and only if the `spender_txhash` key is present.
+
+
+**Result Examples**
+
+::
+
+  {}
+
+::
+
+  {
+    "height": 1866594
+  }
+
+::
+
+  {
+    "height": 1866594,
+    "spender_txhash": "4a19a360f71814c566977114c49ccfeb8a7e4719eda26cee27fa504f3f02ca09",
+    "spender_height": 0
+  }
+
+**Notifications**
+
+  The client will receive a notification when the `status` of the outpoint changes.
+  The protocol does not guarantee but the client might also receive a notification
+  if the status does not change but there was a reorg.
+  Its signature is
+
+    .. function:: blockchain.outpoint.subscribe([tx_hash, txout_idx], status)
+       :noindex:
+
+blockchain.outpoint.unsubscribe
+=================================
+
+Unsubscribe from a transaction outpoint (TXO), preventing future notifications
+if its `status` changes.
+
+**Signature**
+
+  .. function:: blockchain.outpoint.unsubscribe(tx_hash, txout_idx)
+  .. versionadded:: 1.5
+
+  *tx_hash*
+
+    The TXID of the funding transaction as a hexadecimal string.
+
+  *txout_idx*
+
+    The output index, a non-negative integer.
+
+**Result**
+
+  Returns :const:`True` if the outpoint was subscribed to, otherwise :const:`False`.
+  Note that :const:`False` might be returned even for something subscribed to earlier,
+  because the server can drop subscriptions in rare circumstances.
+
+
 blockchain.transaction.broadcast
 ================================
 
@@ -625,15 +744,17 @@ and height.
 
 **Signature**
 
-  .. function:: blockchain.transaction.get_merkle(tx_hash, height)
+  .. function:: blockchain.transaction.get_merkle(tx_hash, height=None)
+  .. versionchanged:: 1.5
+     *height* argument made optional (previously mandatory)
 
   *tx_hash*
 
     The transaction hash as a hexadecimal string.
 
   *height*
 
-    The height at which it was confirmed, an integer.
+    Optionally, the height at which it was confirmed, an integer.
 
 **Result**
 
@@ -968,6 +1089,7 @@ server.version
 ==============
 
 Identify the client to the server and negotiate the protocol version.
+This must be the first message sent on the wire.
 Only the first :func:`server.version` message is accepted.
 
 **Signature**

electrumx/lib/util.py

@@ -34,6 +34,7 @@
 import sys
 from collections.abc import Container, Mapping
 from struct import Struct
+from typing import Optional
 
 
 # Use system-compiled JSON lib if available, fallback to stdlib
@@ -162,7 +163,7 @@ def chunks(items, size):
         yield items[i: i + size]
 
 
-def resolve_limit(limit):
+def resolve_limit(limit: Optional[int]) -> int:
     if limit is None or limit < 0:
         return -1
     assert isinstance(limit, int)
@@ -316,6 +317,7 @@ def protocol_version(client_req, min_tuple, max_tuple):
 struct_le_Q = Struct('<Q')
 struct_be_H = Struct('>H')
 struct_be_I = Struct('>I')
+struct_be_Q = Struct('>Q')
 structB = Struct('B')
 
 unpack_le_int32_from = struct_le_i.unpack_from
@@ -329,6 +331,7 @@ def protocol_version(client_req, min_tuple, max_tuple):
 unpack_le_uint32 = struct_le_I.unpack
 unpack_le_uint64 = struct_le_Q.unpack
 unpack_be_uint32 = struct_be_I.unpack
+unpack_be_uint64 = struct_be_Q.unpack
 
 pack_le_int32 = struct_le_i.pack
 pack_le_int64 = struct_le_q.pack
@@ -337,6 +340,7 @@ def protocol_version(client_req, min_tuple, max_tuple):
 pack_le_uint64 = struct_le_Q.pack
 pack_be_uint16 = struct_be_H.pack
 pack_be_uint32 = struct_be_I.pack
+pack_be_uint64 = struct_be_Q.pack
 pack_byte = structB.pack
 
 hex_to_bytes = bytes.fromhex