What is OSINT (Open Source Intelligence)?

Introduction

The first step in a targeted attack – or a penetration test or red team activity – is gathering intelligence on the target. While there are ways and means to do this covertly, intelligence gathering usually starts with scraping information from public sources, collectively known as open-source intelligence or OSINT. Thanks to social media and the prevalence of online activities, there is such a wealth of legally collectible OSINT available now that this may be required to give an attacker everything they need to profile an organization or individual successfully.

In this post, we’ll get you up to speed on what OSINT is all about and how you can learn to use OSINT tools to understand your own digital footprint better.

What is OSINT?

Suppose you’ve heard the name but are wondering what it means. In that case, OSINT stands for open source intelligence, which refers to legally gathered information about an individual or organization from free, public sources. In practice, that tends to mean information found on the internet. Still, any public information falls into the category of OSINT, whether it’s books or reports in a public library, articles in a newspaper, or statements in a press release.

OSINT also includes information that can be found in different media types. Though we typically consider it text-based, information in images, videos, webinars, public speeches, and conferences all fall under the term.

OSINT is different from other forms of intelligence gathering in several ways, including the following:

OSINT is focused on publicly available and legally obtainable information, whereas other forms of intelligence gathering may involve confidential or classified sources.
OSINT uses various sources, including social media, news articles, public records, and government reports. In contrast, other forms of intelligence gathering may focus on a specific source type.
OSINT often involves using advanced analytical techniques, such as natural language processing and machine learning, to extract insights and intelligence from large volumes of data. In contrast, other forms of intelligence gathering may rely more on human analysis and interpretation.

What is OSINT Used For?

By gathering publicly available sources of information about a particular target, an attacker – or friendly penetration tester – can profile a potential victim to understand its characteristics better and narrow the search area for possible vulnerabilities. Without actively engaging the target, the attacker can use the intelligence produced to build a threat model and develop a plan of attack. Targeted cyber attacks, like military attacks, begin with reconnaissance, and the first stage of digital reconnaissance is passively acquiring intelligence without alerting the target.

Gathering OSINT on yourself or your business is also a great way to understand what information you are gifting potential attackers. Once you know what kind of intel can be gathered about you from public sources, you can use this to help you or your security team develop better defensive strategies. What vulnerabilities does your public information expose? What can an attacker learn to leverage in a social engineering or phishing attack?

What are OSINT Best Practices?

Some best practices for OSINT include:

Develop a clear and comprehensive OSINT strategy: Organizations should develop a clear and comprehensive OSINT strategy that outlines the objectives, goals, and priorities of their OSINT efforts, as well as the specific sources, techniques, and tools that will be used.
Follow legal and ethical guidelines: Organizations should ensure that their OSINT efforts follow relevant legal and ethical guidelines, such as privacy laws and regulations.
Use a variety of sources and techniques: Organizations should use a variety of sources and techniques to gather OSINT, including social media, news articles, public records, and government reports, as well as advanced analytical techniques, such as natural language processing and machine learning.
Ensure the quality and reliability of OSINT: Organizations should take steps to ensure the quality and reliability of their OSINT, such as verifying the accuracy and credibility of sources and conducting regular assessments of their OSINT processes and practices.
Protect the confidentiality and integrity of OSINT: Organizations should implement appropriate measures to protect the confidentiality and integrity of their OSINT, such as encrypting data, securing access to systems and networks, and regularly backing up data.

Overall, following these best practices can help organizations to effectively and efficiently gather, analyze, and disseminate OSINT, while ensuring compliance with legal and ethical guidelines.

Recommended OSINT Tools for Security Research

Many different OSINT (Open-Source Intelligence) tools are available for security research. Some of the most popular and effective tools include:

Maltego: This tool is used for conducting open-source intelligence and forensic analysis. It allows users to collect, visualize, and analyze data from various sources, including social media, the deep web, and other online sources.
FOCA: This tool is used for metadata analysis, allowing users to extract hidden information from documents and other files. It can uncover hidden data, such as IP addresses, email addresses, and other sensitive information.
Shodan: This tool is used for internet scanning and search, allowing users to discover connected devices and networks. It can be used to identify vulnerabilities and potential security threats.
TheHarvester: This tool is used for collecting email addresses, subdomains, and other information from a variety of online sources, including search engines, social media, and the deep web.
Recon-ng: This tool is used for web reconnaissance, allowing users to gather information from various online sources, including social media, DNS records, and the deep web.

These are just a few examples of OSINT tools that can be used for security research. There are many other tools available, and the best one for a given situation will depend on the specific needs and goals of the researcher.

What Are OSINT Skills?

OSINT skills are the abilities and knowledge necessary to collect, analyze, and use information from open sources for various purposes. These skills can be applied in fields such as intelligence, security, and law enforcement, as well as in other areas where access to information is important. Some of the key OSINT skills include:

Understanding the different types of open sources, including public websites, social media, and other online sources.
Knowing how to access and use various OSINT tools and techniques, such as search engines, social media scraping, and metadata analysis.
Developing the ability to analyze and interpret data from open sources, including identifying patterns, trends, and connections.
Building a network of contacts and sources who can provide valuable information and insights.
Having the ability to present findings and conclusions in a clear, concise, and persuasive manner.

Overall, OSINT skills involve a combination of technical knowledge, analytical ability, and interpersonal skills. These skills are essential for anyone working in a field that relies on open-source intelligence.

Do Hackers use OSINT?

Yes, hackers often use OSINT techniques to gather information about potential targets. OSINT involves using publicly available information from social media, websites, and news articles to gather information about an individual or organization. This information can then be used to identify vulnerabilities and plan attacks. Some common OSINT techniques include using search engines to find sensitive information, using social media to gather personal information about an individual, and using public databases to find information about an organization’s employees or infrastructure.

How Can I Use OSINT to Protect my Network?

OSINT can be used to protect networks in a variety of ways, including the following:

Identifying potential threats: Organizations can identify threats, such as new vulnerabilities or emerging attack techniques, by analyzing publicly available information. This can help organizations proactively protect their networks and systems and to stay ahead of potential threats.
Conducting risk assessments: OSINT can gather information on an organization’s operations, assets, and employees, allowing organizations to conduct thorough risk assessments and identify potential vulnerabilities or weaknesses in their networks.
Monitoring public sentiment: By monitoring social media and other online platforms, organizations can gain insights into public sentiment and perceptions of their brand, products, and services. This can help organizations to identify potential issues or concerns and respond to them in a timely and effective manner.

Overall, OSINT can provide valuable information and insights to help organizations better protect their networks and systems from potential threats.

What is the OSINT Framework?

Gathering information from a vast range of sources is time-consuming, but there are many tools to simplify intelligence gathering. While you may have heard of tools like Shodan and port scanners like Nmap and Zenmap, the full range of tools is vast. Fortunately, security researchers themselves have begun to document the tools available.

A great place to start is the OSINT Framework put together by Justin Nordine. The framework provides links to a large collection of resources for a huge variety of tasks from harvesting email addresses to searching social media or the dark web.

In many articles on OSINT tools, you’ll see references to one or two packages included in the Kali Linux penetration testing distribution, such as theHarvester or Maltego, but for a complete overview of available OSINT tools available for Kali, check out the Kali Tools listing page, which gives both a rundown of the tools and examples of how to use each of them.

Among the many useful tools you’ll find here for open-source intelligence gathering are researcher favorites like Nmap and Recon-ng. The Nmap tool allows you to specify an IP address and determine what hosts are available, what services those hosts offer, the operating systems they run, what firewalls are in use and many other details.

Recon-Ng is a tool written in Python by Tim Tomes for web reconnaissance. You can use it to enumerate the subdomains for a given domain, but dozens of modules allow you to hook into things like the Shodan internet search engine, Github, Jigsaw, Virustotal, and others once you add the appropriate API keys. Modules are categorized into groups such as Recon, Reporting, and Discovery modules.

Other OSINT Tools, Techniques, and Resources

One of the most prominent tools for use in intelligence gathering is, of course, web search engines like Google, Bing and so on. In fact, there are dozens of search engines, and some may return better results than others for a particular kind of query. The problem is, how can you efficiently query these many engines?

A great tool that solves this problem and makes web queries more effective is Searx. Searx is a metasearch engine that allows you to anonymously and simultaneously collect results from more than 70 search services. Searx is free; you can even host your own instance for ultimate privacy. Users are neither tracked nor profiled, and cookies are disabled by default. Searx can also be used over Tor for online anonymity.

Many public instances of Searx are also available for those who either don’t want or don’t need to host their own instance. See the Searx wiki for a listing.

There are many people working on new tools for OSINT all the time, and a great place to keep up with them and just about anything else in the cybersecurity world is, of course, by following people on Twitter. Keeping track of things on Twitter, though, can be difficult. Fortunately, an OSINT tool for that, too, is called Twint.

Twint is a Twitter scrapping tool written in Python that makes it easy to anonymously gather and hunt for information on Twitter without signing up to the Twitter service itself or using an API key as you would have to do with a tool like Recon-ng. With Twint, there’s no authentication or API needed at all. Just install the tool and start hunting. You can search by user, geolocation and time range, among other possibilities. Here are some of Twint’s options, but many others are available.

So how can you use Twint to help you keep up with developments in OSINT? Well, that’s easy and is a great example of Twint in action. As Twint allows you to specify a --since option to only pull tweets from a certain date onwards, you could combine that with Twint’s search verb to scrape new tweets tagged with #OSINT on a daily basis. You could automate that script and feed the results into a database to view at your convenience by using Twint’s --database option that saves to SQLite format.

Looks like there’s been 58 #OSINT tweets so far today!

twint -s '#osint' --since 2019-07-17

Another great tool you can use to collect public information is Metagoofil. This tool uses the Google search engine to retrieve public PDFs, Word Documents, Powerpoint and Excel files from a given domain. It can then autonomously extract metadata from these documents to produce a report listing information like usernames, software versions, servers and machine names.

Conclusion

In this post, we’ve covered the basic idea of OSINT and why it’s useful. We’ve looked at a couple of great places where you can discover many OSINT tools to help you with virtually any information gathering you need. We’ve also given you a taste of a few individual tools and shown how they can be put to work.

Understanding how to collect open-source intelligence is vital for anyone involved in cybersecurity. Whether you’re defending an enterprise network or testing it for weaknesses, the more you understand its digital footprint, the better you can see it from an attacker’s point of view. Armed with that knowledge, you can then go on to develop better defensive strategies.

Schedule A Demo

SentinelOne encompasses AI-powered prevention, detection, response and hunting.

Get A Demo

Like this article? Follow us on LinkedIn, Twitter, YouTube or Facebook to see the content we post.