Understanding the Importance of Threat Intelligence Data Collection

The collection of threat data is one of the most crucial stages, if not the most crucial, of the threat intelligence lifecycle. The quality of the data collected at this stage will define all the following stages.

With low-quality, inaccurate, or undiversified data, the subsequent analysis will produce inaccurate results, leading to ineffective or even potentially harmful actions. 

Data diversity is key

At the data collection stage, threat intelligence analysts should aim to gather data from a great variety of high-quality sources and utilize diverse collection methods, including automated data feeds, manual research, threat hunting, and collaboration with other organizations to share threat information.

Let’s take a closer look at the different threat data sources analysts should leverage.

Open Source Intelligence 

Never underestimate or ignore the power and knowledge the open source and hacker communities hold. Open source intelligence (OSINT) provides analysts with valuable information gathered from publicly available sources such as news articles, social media, forums, blogs, and websites. OSINT provides valuable insights into emerging threats, cybersecurity trends, and hacker chatter.

Security teams can leverage OSINT tools and techniques to monitor mentions of their brand, industry-specific keywords, or Indicators of Compromise (IoCs) linked to known threats.

OSINT is particularly useful for identifying early warning signs of potential cyber attacks, gathering threat intelligence on specific threat actors or campaigns, and understanding the broader threat landscape.

Internal security tools and logs

Another invaluable source of threat information comes from internal security tools and logs, including Intrusion Detection Systems (IDS), Endpoint Detection and Response (EDR) solutions, Security Information and Event Management (SIEM) systems, and log files from the various infrastructure components.

Internal security tools generate data on network traffic, system events, user activity, and other aspects of the organization’s IT environment. Analyzing this data can help identify signs of malicious activity, unusual behavior, or security incidents.

Integrating those internal security tools with external threat intelligence feeds enables analysts to correlate internal events with external threat intelligence, enhancing the overall threat detection and response capabilities of security teams.

Commercial threat intelligence feeds

Security teams often subscribe to commercial threat intelligence feeds provided by cybersecurity vendors, research firms, and industry-specific information-sharing groups. These feeds deliver timely and relevant threat intelligence tailored to the organization’s needs.

Commercial threat intelligence feeds aggregate data from a wide range of sources, including proprietary research, malware analysis, dark web monitoring, and global threat intelligence networks. However, you should make a point to verify the source of the data provided by the vendor. 

More often than not, commercial intelligence feeds rely on data scraped from not-so-reliable third-party sources or honeypot networks. While these sources may provide a great volume of data, the quality is often low.  Also, it is important to mention that honeypots are easily detected by attackers who know how to avoid them and preserve their IP addresses.

High-quality and accurate intelligence feeds include data from real users, real servers, and real production environments. These feeds provide analysts with valuable information on IOCs, malware signatures, threat actor profiles, and other actionable intelligence to help security teams detect and respond to cyber threats more effectively.

Government and law enforcement agencies

Government agencies and law enforcement organizations can also be a source of valuable threat information. They collect data on cyber threats through various means, including cyber threat assessments, incident reports, and collaboration with international partners, and often share that information with the private sector.

Security analysts can access threat data from government sources such as the FBI, , Department of Homeland Security (DHS), National Security Agency (NSA), European Union Agency for Cybersecurity (ENISA), and international organizations like INTERPOL and Europol.

Government threat intelligence often includes information on nation-state actors, Advanced Persistent Threats (APTs), cyber espionage, and other high-profile cyber threats with national security implications.

Information Sharing and Analysis Centers & Information Sharing and Analysis Organizations

Information Sharing and Analysis Centers (ISACs) and Information Sharing and Analysis Organizations (ISAOs) are industry-specific, non-profit organizations that facilitate the sharing of threat intelligence and best practices among member organizations. Organizations like the U.S. Cybersecurity and Infrastructure Security Agency (CISA) or the Financial Services ISAC (FS-ISAC), collect and disseminate threat information related to specific sectors such as finance, healthcare, energy, and transportation.

These organizations serve as hubs for collaboration and information sharing, enabling members to exchange threat intelligence, incident reports, and defensive strategies to improve cybersecurity posture collectively.

Participating in ISACs and ISAOs allows security teams — and their organizations as a whole — to access timely and relevant threat intelligence specific to their industry vertical, enhancing their ability to detect and respond to sector-specific threats.

Unveiling the truth behind threat intelligence data

As the popularity of Digital Risk Protection Services (DRPS) and External Attack Surface Management (EASM) continues to boom, the reliance on credible threat intelligence data is critical. Gartner’s recent Market Guide for Security Threat Intelligence Products and Services highlights the convergence of these services with threat intelligence, emphasizing the need for a multi-sourced, well-integrated, and automated approach to threat detection and response.

The CrowdSec model is a testament to this need for reliability and transparency in threat intelligence. With a community-driven approach, CrowdSec ensures the data powering DRPS and EASM solutions is both comprehensive and trustworthy. 

Our consensus algorithm and trust scoring system are unique methods that vet and refine threat data, creating a secure and transparent cybersecurity environment. While we may stand alone in this endeavor, we commit to maintaining transparency and publishing details about our data curation because we believe this level of transparency should be demanded and expected by end-users.

Vendors in the DRPS and EASM spaces, the message is clear: the quality of your services is contingent upon the integrity of your threat intelligence data. In cybersecurity, opacity is not an option. Transparency in data collection and processing is not merely an additional benefit — it’s the essential foundation that ensures clients’ decisions are well-informed and secure, thereby reducing the risk of significant impacts from cyber threats. 

This cooperative and explicit approach is what will characterize the future of robust cybersecurity strategies.

End-users, it is crucial to demand greater transparency regarding the origins of your vendors’ data. Given the escalating severity of threats and the imminent rise of offensive AI, it is no longer sufficient to be reassured by a brand, regardless of its prominence. 

Finding reliable information on the origins of threat intelligence data is challenging in 2024, and this opacity is no longer tolerable. Threat intelligence entities must now make a concerted effort to demystify the origins of their data.

Since 2019, CrowdSec has set a precedent by publishing all its sources under the MIT license, distinguishing itself as the sole threat intelligence provider to disclose its curation methodology publicly.

Intrigued? You can find the detailed information on how we collect, validate, and curate our threat intelligence data, including our sources and methodologies, publicly available on this website. Take a look!