URL Hash List

The URL Hash List API provides a robust solution for retrieving all MD5 URL hashes stored within the TCAP database. This powerful tool is designed to address the challenge of ensuring that no link to a specific piece of content is overlooked.

Transforming URLs into Working Variations

At the core of this API's functionality is the innovative process of transforming each incoming URL into up to 13 potential working variations. This is achieved through the application of a sophisticated algorithm that generates these alternatives, which are then hashed using the industry-standard MD5 algorithm.

Comprehensive URL Hash Retrieval

By leveraging this API, users can effortlessly retrieve the complete collection of MD5 URL hashes from the TCAP database. This comprehensive data set ensures that even the most obscure or hidden links are surfaced, enabling users to maintain a complete and up-to-date record of all terrorist-content-related URLs.

By harnessing the capabilities of the URL Hash List API, you can unlock the full potential of your content, ensuring that no link is left behind and that every piece of information is readily accessible.

In order to use the URL Hash List you will need to be an on-boarded URL Hash List TCAP user. If you're interested in working with us please get in touch.

URL Hash List

Once you're authenticated you can use your JWT to access the Content Hash List

Open API references:

• redoc
• swagger

Python

import requests

url = "https://beta.terrorismanalytics.org/core/url-hash-list"

headers = {
    "Authorization": f'Bearer {token}"
    "Content-Type": "application/json"
}

response = requests.get(url, json=payload, headers=headers)
print(response.json())

TypeScript

const url = "https://beta.terrorismanalytics.org/core/url-hash-list"

const makeRequest = async (url: string) => {
  const response = await fetch(url, {
    method: 'GET',
    headers: {
      'Authorization': `Bearer ${token}`,
      'Content-Type': 'application/json'
    },
  });

  const data = await response.json();
  console.log(data);
};
const result = await makeRequest(url)

cURL

curl -X GET \
  https://beta.terrorismanalytics.org/core/url-hash-list \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \

Query Parameters

Parameter	Type	Description
ideology	"islamist | far-right"	Specifies the ideology to filter the results
from	"<YYYY-MM-DD>"	Query start date
to	"<YYYY-MM-DD>"	Query end date
limit	Number (max 1000)	N entries per response
offset	Number	How many entries to skip
count	Number	Total number of hashes in the archive when the request was made
next	String	Endpoint to retrieve the next page
previous	String	Endpoint to the previous page
results	Array	List of hashes with all variations

Response

The Hash List endpoint returns a paginated response of URL Hash objects, where each object contains a collection of URL hash digests.

Field	Type	Example Value
count	Number	56498
next	String	"http://beta.terrorismanalytics.org/core/url-hash-list?limit=10&offset=10"
previous	String|null	null
results	Array<Result>	Array of results (see Result object structure below)

Result object structure:

Field	Type	Example Value
id	Number	2
updated_on	String	"2024-12-17T18:29:33.030336"
hashes	Array<Hash>	Array of hash objects (see Hash object structure below)

Hash object structure:

Field	Type	Example Values
hash_digest	String	"2908abcc1b389b20668c9003a51a7ae3"
hash_type	String	"MD5" or "MDL"
normalisation_methods	Array<String>	[], ["LOWERCASE_ORIGIN", "ADD_HTTP_PROTOCOL"], etc.

Example response

{
  "count": 56498,
  "next": "http://beta.terrorismanalytics.org/core/url-hash-list?limit=1&offset=1",
  "previous": null,
  "results": [
    {
      "id": 2,
      "updated_on": "2024-12-17T18:29:33.030336",
      "hashes": [
        {
          "hash_digest": "<000aaa111bbb222ccc333ccc444ddd555>",
          "hash_type": "MD5",
          "normalisation_methods": []
        },
        {
          "hash_digest": "aaa111bbb222ccc333ccc444ddd555eee",
          "hash_type": "MD5",
          "normalisation_methods": [
            "UPPERCASE_ORIGIN",
            "ADD_HTTP_PROTOCOL",
            "ADD_WWW_PREFIX"
          ]
        },
        {
          "hash_digest": "111bbb222ccc333ccc444ddd555eee666",
          "hash_type": "MD5",
          "normalisation_methods": ["UPPERCASE_ORIGIN"]
        }
      ]
    }
  ]
}

Open API references:

• redoc
• swagger