Request for Assistance with Shodan API 'facet' Timeout Issue #240
Description
Hello, First of all, I wanted to apologize for having to create an issue on this GitHub project for this topic.
It may seem out of scope since it concerns a server-side issue at Shodan and therefore directly the Shodan Python SDK/Wrapper/Library.
I'm creating this issue following an attempt to contact Shodan via:
- Email ([email protected])
- Twitter (@shodanhq)
sent on 05/11/2025.
In the absence of a response, I'm copying and pasting the message:
I am reaching out to you regarding an issue I am encountering with the Shodan API, specifically with the 'facet' functionality. I am trying to retrieve a large list of "CPE" entries based on product filters such as "product:openssh". However, when I execute this query, whether through your Web UI, the Shodan CLI, or the Shodan Python module, I receive the following error:
shodan.exception.APIError: The search request timed out or your query was invalid.
You can verify this behavior yourself using the following URL:
https://www.shodan.io/search/facet?query=product%3Aopenssh&facet=cpe
I am particularly interested in retrieving this CPE list for a specific reason. An analysis of the CPEs provided by your service suggests that you are using the Nmap tool, specifically the signature database available here:
https://svn.nmap.org/nmap/nmap-service-probes
However, for a service like OpenSSH, this database includes CPEs like:
cpe:/a:openbsd:openssh:6.0p1
Yet, this particular CPE does not exist in the NIST database. Instead, we have the following valid entries:
cpe:/a:openbsd:openssh:6.0
cpe:/a:openbsd:openssh:6.0:p1
It is worth noting that for OpenSSH, the code for versions 6.0p1 and 6.0 is identical (as confirmed by OpenSSH commit history).
As a result, my goal is to retrieve a comprehensive list of CPEs for the services present in the nmap-service-probes file, so I can verify whether they exist in the NIST database. If not, I would like to adjust the regular expressions in the nmap-service-probes file to ensure the CPEs reported are consistent with the NIST database.
Would you be able to assist me in resolving this timeout issue or alternatively provide guidance on obtaining this CPE list?
For your reference, here is a list of 'products' extracted from the CPE fields I found in the nmap-service-probes file (covering both applications and operating systems):
ios, backup_server, windows, postgresql, busybox, linux_kernel, os_400, openvms, tftp_server_sp, dropbear_ssh_server, platinum, jboss_enterprise_web_platform, mqx, windows_services_for_unix, fedora, sql_server, httpd, evolution_media_server, isa_server, cassini, easy_file_sharing_ftp_server, exceed, whatsup, imail, sendmail, hp-ux, http_server, openssl, linux_imaging_and_printing_project, cfingerd, debian_linux, realsecure_workgroup_manager, ehttp, tftp_server, apt-cacher, 3cx_phonesystem, ux, webcit, mailenable, rompager, dixim_media_player, sco_unix, asp.net, boa, liferay_portal, jboss_wildfly_application_server, anti-virus, ws_ftp, python, webiopi, femitter_server, digifort, telnet, micro_httpd, tomcat, sunos, emweb, thttpd, minidlna, php, minix, openssh, compaqhttpserver, system_management_homepage, xmanager, activesync, cometcatchr, fireware_xtm, radmin, ntop, freebsd, appweb, serv-u, dovecot, ubuntu_linux, monitor_server, ipos, datacap, filenet_system_manager_dashboard, x.org_x11, linux, eft_server, ltib, data_ontap, windows_media_services, procurve_switch_software, ftp_server, vxworks, sshelper, android, bind, xfree86, mandrake_linux, asyncos, proftpd, bbvs, mac_os_x, teamspeak2, ftp_service, netware, dc_software, miniupnpd, dnas, fortress_ssh_server, cyrus_imap_server, iphone_os, os2_ftp_server, os2, vicftps, hasp_license_manager, ftp_print_server, windows_vista, ftpd, universal_media_server, goahead_webserver, qotd, ssh, rocket, smartertools_web, commerce_server, windows_server_2012, lighttpd, software_data_cable, eyeconnect, windows_10, aix, exchange_server, supportworks_itsm, twonky, application_and_content_networking_system_software, mobaxterm, antivirus, antivirus_scan_engine, ip_sharer_web, webdavserver, cherokee, twisted, twistedweb, abyss_web_server_x1, $SUBST(1,"-","_")firmware, plan_9, quicktime_streaming_server, identity_services_engine_software, lua, open_enterprise_server, basehttpserver, apple_tv, typsoft_ftp_server, webshield_smtp, j2ee_engine, netweaver, ssh_server, node.js, xen_orchestra, freefloat_ftp_server, fuppes, ns-slapd, directory_server, go, influxdb, multi_server, tightvnc, shareport_web_access, mongodb, pycharm, mdaemon, cesarftpd, univention_corporate_server, zftpserver, afp_server, irix, easy_file_management_web_server, filemanager, junos, dns, windows_server_2008, groupwise, phpstorm, java_profiler, ultravnc, activemq, smarterstats, ability_mail_server, quasarrat, rtos, telnetd, informix_dynamic_server, filecopa, ios_xr, simpleserver_www, mini_httpd, print_server_web, winproxy, tmeter, a_ux, zoiper_for_windows, adaptive_server, titan_ftp_server, cherrypy, wsgidav, amx_firmware, memcached, jre, targetweb, netatalk, netmeeting, vm_virtualbox, aironet_350, vcxsrv, filezilla_server, olimpoks, secure_access_control_server, glassfish_server, jsp, timos, lucene, elasticsearch, axigen_mail_server, tivoli_storage_manager, windweb, bqtftp, $1, windows_nt, endpoint_protection_manager, secure_gateway, extremeware, storageworks_secure_path, monkey_http_daemon, wakanda_application_framework, wakanda_server, slackware_linux, radicale, openbsd, traffic_inspector, matrixssl, nginx, jetty, easerver, payara, securityspy, netkit, ftp_server_ultimate, arch_linux, windows_ce, completeftp, softphone, $2, hotel, freesshd, durian_web_application_server, sofia-sip, httrack, forefront_threat_management_gateway, lotus_domino_web_server, application_server_web_cache, ray_server_software, smartware, aos, asterisk, freepbx, realsecure_server_sensor, iis, ioscallrecorder, wsgiref, utorrent, wifi_file_transfer, epolicy_orchestrator_agent, synchronet, qmail, printopia, directadmin, ability_ftp_server, uhttpd, access_gateway, winsshd, kerberos, sgos, meteor, rocket.chat, mailgate, tru64, fedora_core, cygwin, windows_server_2003, openmanage_server_administrator, mercur, embperl, cadvisor, mailtraq, pcanywhere, windows_7, hotspot, routeros, teamspeak_classic, winpcap, enterprise_linux, windows_xp, virtual_server, inn, sentinel_protection_installer, digital_unix, enterprise_security_suite, zlib, broadcam, zenworks_patch_management_server, psftpd, developer_tools_for_upnp, m5t_sip_client_engine, systems_management_server, plex_media_server, xsp, smartermail, hydra, warftpd, 4d, topix8, ichain, chip-rtos, httpfileserver, cockpit, realplayer_cloud, streamwhatyouhear, cuteftp, postfix, sametime, mail_security, zope, zope_enterprise_objects, corrad, easy_file_sharing_web_server, silvershield, airdroid, growl, nut_os, neptune, spybot_search_and_destroy, eweb, rl-arm, sql_monitor, pcman%27s_ftp_server, trillian, coyote_http_connector, my_net_firmware, winroute, xmail, nvision, nsure_audit, lotus_protector_for_mail_security, ftprush, .net_framework, blackberry_playbook_os, aiohttp, flash_media_server, wowza_media_server, filetransfer, home_ftp_server, turboftp, simplehttpserver, kace$1_systems_management_appliance_software, os_390, tornado, xming, zos_explorer, z%2fos, windows_server_2016, hardware_management_console, netbsd, magicmail, vyatta_vrouter_software, qnx, openfire, mac_os, windows_2000, nvidia_streamer_service, advanced_lights_out_manager, opensuse, ruby, weblogic_server
If you ever decide to close this issue without a response, I would completely understand, but could you please escalate the problem to your teams?
Best regards,