All thanks to dark web cartography, we now see how the darknet ecosystem has become the classic "Wild West" - an online decentralized environment that is filled with a rather diverse user base.
The existing law enforcement agencies have strived hard to contain the illicit factions of darknet use.
While cross-border authorities are in a constant endeavor to bring dark web activities to book, much is still at stake as far as the entire internet underworld is concerned.
Cybersecurity establishments have realized the limitations of relying on traditional undercover and sting operations targeting the dark web.
This factor stems from the attributes of the ecosystem being highly dynamic, developing fool-proof guises on a continual basis-and thus forcing the relevant law enforcement agencies to expend massive resources in policing darknet websites.
The above mentioned issues have created room for the application of crawling and scraping techniques as reliable means to investigating online corridors.
This, in combination with darknet cartography research, helps investigators better understand how the ecosystem works.
What Exactly Is Hidden Web Crawling?Put simply, the practice of dark web crawling involves the identification of emergent onions within darknet spaces on appearance, including the aspect of indexing these sites to activate cybersecurity measures involved in the algorithmic isolation and analysis of online threats.
Dark web/deep web crawlers convert dark web sites into machine-discernible data that would enable cyber intelligence experts to pinpoint potential patterns and traces of illicit darknet and deep web activities that typify online marketplaces.
Such action is usually followed by the destruction of these elements.
The Essence of Crawling the Deep and the Dark Web
Before delving into the process of web crawling, it is important to first understand the significance of integrating crawling technologies in targeting the deep and dark web ecosystem.
Early Detection of Security Vulnerabilities
First, web crawling can be instrumental in the prevention of potential cyberattacks. The dark web offers ample space for the trade of malware.
It is in this particular environment that most new malware programs are first released before being used to attack otherwise "clean" computer systems.
Take the example of a 2015 case that involved the identification of a serious Windows OS vulnerability by Microsoft.
Although the corporation managed to provide the much-needed fix in good time, elements of the vulnerability had already been spread through the dark web accordingly-in form of the infamous Dyre Banking Trojan.
The Dyre Banking Trojan targeted users across the globe to harvest their credit card information.
It is from this case that experts from Arizona State University partnered in creating a web crawling mechanism that would stalk dark web forums to identify security vulnerabilities (see below).
The above example justifies the need to acknowledge the fact that hackers have always used the dark web as a reliable vehicle for undertaking their attacks-thus calling for the mobilization of potent dark web crawling resources.
Though distinct, real world and dark web crimes are often relatable within the spheres of global cybersecurity.
Web crawling strategies have become the staple of various organizations for upholding competitive corporate intelligence with the support of thorough web research.
It is agreeable that crawlers can be used in the context of darknet applications, and those of the deep web, as an intervention to tackle crimes occurring on the dark web.
Still, the elements of real-world criminal enterprises come into play whenever law enforcement considers the need to keep darknet platforms in check.
Conventional crime follows clear operational paths, including the reliance on darknet markets to orchestrate, delegate and procure illicit goods and services.
The existence of terrorist platforms in the dark web, for example, presents significant threats to national, regional and international security.
However, terrorist attacks may be prevented in good time whenever cybersecurity frameworks employ data mining and predictive modelling methods to scan darknet forums.
Web crawlers offer endless possibilities in terms of what can be done to detect hints concerning the planning of terrorist attacks-in which case the relevant law enforcement agencies may step in to prevent such criminal events.
Understanding Web Crawlers
Web crawlers are designed to automatically analyze, process and engage pages and search forms.
These functions are automated by various crawling tools in different ways.
In principle, crawling the web requires a dedicated task of discovering resources and extracting content.
In the context of resource discovery, a crawler searches for target websites that bear certain attributes.
On the other hand, content extraction involves the obtainment of information from these online destinations by filling out entries that represent certain presets or keywords.
The basic strategy of a deep/dark web crawler is similar to those of other conventional web crawlers, such as Google's system.
A traditional crawler chooses URL sources, recovers webpages and processes them before extracting relevant links.
The only distinguishing factor between traditional and hidden web crawlers is that, while traditional crawlers fail to distinguish between pages (with or without forms), darknet crawlers execute sets of actions to target each form existing in a page.
Thus, the following steps describe the typical process undertaken by hidden web crawlers;
A search form extraction procedure takes place, in which relevant tags in webpages are found.
The crawler analyzes and digests the page form to construct an internal form presentation.
The crawler uses an approximate string matching between the form labels, including those embedded in the database to establish a class of value.
At this point, the crawler submits the filled form to the web server.
Finally, a response is analyzed to ascertain the validity of search results as per respective submission. Hypertext links are thus crawled in the response page accordingly.
An important point to note is that while the above steps describe the general process of hidden web crawling, it is possible that tech experts may tailor new crawling techniques to suit specified needs-such as this new hidden web approach that's designed to be domain-specific.
Otherwise, in general, hidden web crawlers are methodologically classified as follows;
In this classification, deep web crawling works by crossing a broad variety of URL sources instead of repeatedly crawling content attributed to a limited source of URLs.
These crawlers work by harvesting the maximum possible data load from a specific URL source.
In the sense of keyword selection methods, the following crawler classifications hold true;
Such types of crawlers employ random dictionaries to get the keywords needed to analyze and interact with search forms. It is common that such used dictionaries are tailored for specific web domains.
In this class, crawlers rely on a system of generic distribution as far as the frequencies of keywords are concerned. This method helps in producing resultant dependable matching to save time during the crawling process.
In adaptive keyword selection methods, crawlers analyze data obtained from shortlisted search keywords that provide maximal content. Such shortlisted keywords aim to maximize on the quality of output concerning the resultant queries derived from the entire process.
A Specific Example: ACHE
As mentioned already, hidden web sites cannot be crawled by traditional crawlers owing to the fact that web servers utilize anonymity of the Tor network.
ACHE is a focused crawler that gathers webpages that meet specific requirements.
Such provisions range from a sense of belonging to particular domains or pages that follow a prescribed pattern.
The uniqueness of ACHE is brought about by its page classifier feature, which filters pages that occur within a certain domain.
Simply, page classifiers operate within the precincts of machine learning to match pages that are associated with target words.
Additionally, ACHE bears an automated ability to learn link prioritization in order to smoothly pinpoint relevant content without the harvesting of unwanted pages.
In the specific context of darknet browsing, ACHE utilizes external HTTP proxies to route requests to .onion addresses residing on Tor.
This attribute charges it with the important role of crawling the otherwise hidden deep and dark web sites through a prescribed step-by-step process as shown in the figure below.
Conclusion
In conclusion, like economic systems, the effects of technological advancement on societies is heavily contingent on user morals - while criminals may want to take advantage of digital systems, the same tech tools may prove to be the ultimate antidote to troubled victims.
The improvement of machine learning systems should be seen as a welcome idea as long as safe online activity stands to be the desire of every netizen.
It is this exploration and balances in synergies that illicit activities on the dark web may be contained within safe limits.
