For a project I need to write a scraper that sends a GetAddr request to all Bitcoin nodes. I determine the reachable nodes using the bitnodes.io API. Then I go through all the IPs and send a request using the python-nmap library and the bitcoin-getaddr script from nmap.
The code to do this looks like the following:
def send_get_addr_requests(ip, port): ip_addr = ipaddress.ip_address(ip) # check if it is an ipv6 address and fix arguments in this case arguments_get_addr = '--script bitcoin-getaddr -d -Pn' if isinstance(ip_addr, ipaddress.IPv6Address): arguments_get_addr += ' -6' # send get_addr request to the node nm = nmap.PortScanner() response_get_addr = nm.scan(ip, port, arguments=arguments_get_addr) return response_get_addr
However, as I have found, the nmap script (especially the bitcoin.lua file) is relatively unreliable and in most cases does not return a result on the request. I have already modified the GetNodes function as follows:
getNodes = function(self) local req = Request.GetAddr:new( self.host, self.port, self.lhost, self.lport ) local status, err = self.socket:send(tostring(req)) if ( not(status) ) then return false, "Failed to send \"GetAddr\" request to server" end local status, response = Response.recvPacket(self.socket, self.version) local all_addrs = {} -- limit is not usable because it is unclear how many packets are returned -- thats why we take a timer -- local limit = 20 local startTime = os.time() local endTime = startTime + 60 -- wait 60 seconds max -- Usually sends an addr response with 1 address, -- then some other stuff like getheaders or ping, -- then one with hundreds of addrs. while #all_addrs <= 1 and os.time() < endTime and status do -- limit = limit - 1 status, response = Response.recvPacket(self.socket, self.version) if status and response.cmd == "addr" then for _, addr in ipairs(response.addresses) do all_addrs[#all_addrs+1] = addr end end end return #all_addrs > 0, all_addrs end,I added a timer and removed the limit. Since the requested nodes send many packets before the GetAddr response, this approach works much better. I also set the replaced version to 70015 and adjusted the ping/pong messages of the script to the newest implementation.
However, in many cases (about 2000 out of 7000) I still do not receive a GetAddr response. The function Response.recvPacket(self.socket, self.version) from the bitcoin.lua file gets at some point only a status = false and a response = EOF ...
So my question is if this is normal or how I can further customize the script to get a more reliable response? When I manually type in the terminal "nmap -Pn -d -script bitcoin-getaddr -p 8333 " I get a response from time to time, but not reliable and rather rare. So the requested nodes don't block the GetAddr requests, which makes the whole thing even more complicated to understand.
Thanks in advance for any help!
Here is another Wireshark screenshot. I executed the command several times in a row (14x) in the terminal. In the last attempt I finally got the GetAddr response. The [unknown command] packets are "sendheaders" and "sendcmpct" packets (and in the last attempt "feefilter"). 
You can get bonuses upto $100 FREE BONUS when you:
💰 Install these recommended apps:
💲 SocialGood - 100% Crypto Back on Everyday Shopping
💲 xPortal - The DeFi For The Next Billion
💲 CryptoTab Browser - Lightweight, fast, and ready to mine!
💰 Register on these recommended exchanges:
🟡 Binance🟡 Bitfinex🟡 Bitmart🟡 Bittrex🟡 Bitget
🟡 CoinEx🟡 Crypto.com🟡 Gate.io🟡 Huobi🟡 Kucoin.
Comments