/get_blocks.bin is HTTP binary RPC serialized with portable storage, not a
P2P Levin message. The previously added pool limiter used
LEVIN_DEFAULT_MAX_PACKET_SIZE as a response budget, but that constant does
not apply to the RPC path.
Add an include_sensitive parameter to tx_memory_pool::get_pool_for_rpc
(and its core passthrough), mirroring the include_sensitive_data
parameter on the HTTP analog get_transactions_and_spent_keys_info.
When false, receive_time and last_relayed_time are zeroed using the
same masking already applied on the HTTP path.
The ZMQ handler in daemon_handler.cpp passes !m_restricted, so
--restricted-zmq-rpc callers now receive the same privacy-filtered view
as restricted HTTP callers instead of the unfiltered timing metadata
they previously got. Stem-phase txs continue to be excluded regardless
(relay_category::broadcasted filter unchanged).
Refs #10529.
Co-authored-by: nahuhh
- jberman review
- cryptonote_protocol: don't arbitrarily download 1000 blocks ahead
- further restrict `proceed` to require `queue_proceed` in all cases.
ensure queue_proceed is true if we need the next block, even if we
already exceed the span and size limits
cryptonote_protocol: improved logging + const usage in span downloader
Does the following to speedup the `/is_key_image_spent` RPC endpoint:
- Reads all on-chain key images in one LMDB read transacion
- Uses `cryptonote_core::are_key_images_spent_in_pool()` instead of `cryptonote_core::get_pool_transactions_and_spent_keys_info()` for pool querying. This only does a LMDB read per key image if in the pool.
- Filters known on-chain spent key images before querying for key images spent in pool
This RPC endpoint was causing major daemon slowdowns in the Carrot/FCMP++ Alpha Stressnet when using the `rescan_spent` command, especially for large wallets.
The effect was much worse if the mempool was full.
Without this commit:
1) read height from DB
2) add block to chain in separate thread
3) read chain for block id's and request them from peer
4) ERR in handle_response_chain_entry, peer's first block is the
one that was added to the chain, which has block idx=height from
step 1.
This commit reads the chain for height and highest block id's
in one go while holding the m_blockchain_lock to avoid the race.
Changes complexity from M*N to (2*N+M)*log2(M). The FCMP++ stressnet recently hit mempool sizes of ~55k txs.
If the requesting node's mempool is populated, this results in an average of (55000*55000)/2
(about 1.5 billion) comparisons for the responding node. Under this commit, this would be reduced to
(55000+55000)*log2(55000) comparisons (about 2.6 million), a 99.83% reduction.
When the request includes block_ids, the daemon uses
find_blockchain_supplement to identify the highest block hash
passed in block_ids that the daemon also knows about, and then
serves subsequent blocks contiguous to that block.
When block_ids_skip_exclusive is false (default current
behavior), the daemon includes the highest block requested in the
response, in addition to contiguous blocks after it.
When block_ids_skip_exclusive is true (new param), the daemon
serves blocks starting from the block 1 higher than the highest
known block included in block_ids. Thus, the daemon skips the
common block known to the client and daemon. Clients can make sure
the daemon is serving expected contiguous blocks to its highest
known block by checking the first block's prev_id included in the
response, and making sure it is equivalent to the block the client
already knows about that was included in block_ids. This avoids
the daemon serving 1 extra block it does not need to serve to the
client, since the client should already know about that block.
bl
ffbf9f4 blockchain_and_pool: move to crytonote_core and enforce its usage (jeffro256)
d6f86e5 Avoid nullptr dereference when constructing Blockchain and tx_memory_pool (lukas)
- `/getblocks.bin` respects the `RESTRICTED_TX_COUNT` (=100) when
returning pool txs via a restricted RPC daemon.
- A restricted RPC daemon includes a max of `RESTRICTED_TX_COUNT` txs
in the `added_pool_txs` field, and returns any remaining pool hashes
in the `remaining_added_pool_txids` field. The client then requests
the remaining txs via `/gettransactions` in chunks.
- `/gettransactions` no longer does expensive no-ops for ALL pool txs
if the client requests a subset of pool txs. Instead it searches for
the txs the client explicitly requests.
- Reset `m_pool_info_query_time` when a user:
(1) rescans the chain (so the wallet re-requests the whole pool)
(2) changes the daemon their wallets points to (a new daemon would
have a different view of the pool)
- `/getblocks.bin` respects the `req.prune` field when returning
pool txs.
- Pool extension fields in response to `/getblocks.bin` are optional
with default 0'd values.
update_checkpoints() makes a few DNS requests and can take up to 20-30 seconds to complete (3-6 seconds on average). It is currently called from core::handle_incoming_block() which holds m_incoming_tx_lock, so it blocks all incoming transactions and blocks processing while update_checkpoints() is running. This PR moves it to until after a new block has been processed and relayed, to avoid full monerod locking.