This is the processor of the "Information Manipulation Analyzer" project.
The main idea is to show the volumetry of a keyword,hashtag,cashtag along with some other informations in order to have a better idea of whether or not a narrative has been artificially improved.
The frontend that will actually log the searches to analyze and display the results can be found here
We are using now Snscrape to scrape all the data (twint has been abandoned due to some unstable requests). Plan is to migrate Snscrape to the french sovereign bot whenever it will be ready.
We are also using a Bot detection library created by AmbNum and that needs to be installed
Main program launches concurrently 3 processes to keep some kind of separation of concern in case we want someday to extract some of them into different microservices
src/hashtags.ts
will poll on every tick and retrieve volumetry for one hashtag at a time, beginning with the most recentsrc/users.ts
will poll on every tick and use the bot detection library to enhance user's data in the databasesrc/server.ts
will expose a server that permits to execute the above services with API calls. (This server is not secured as it is only meant to be used locally through http://localhost:4000)src/retweets.ts
will poll on every tick and retrieve all RT since last call and update the given tweets with the latest information
IMPORTANT main branch is main
but all PRs must be against develop
, except for immediate patches
Install mongoDB and lauch instance
brew tap mongodb/brew
brew install mongodb-community@4.4
mongo --version
brew services start mongodb-community
then install mongoDB Compass and create a local database
Create a .env.local
file at the root of the project (You can copy it from .env.local.example
)
and don't forget to change the name of your local database on the MONGODB_URI
value
NODE_PATH="src"
MONGODB_URI="mongodb://localhost:27017/database-name?&compressors=zlib&retryWrites=true&w=majority"
Install snscrape
pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git
Install jq (for formatting json)
brew install jq
Then launch
yarn
yarn dev
In order to use the bot detection functionnality, you need to get a provider.
We have created an open source one which we can use. See social-networks-bot-finder
In order to impletement it you need to install it somewhere on your machine and add to .env.local
two environment variables
BOT_SCORE_PROVIDER=social-networks-bot-finder
BOT_SCORE_SOCIAL_NETWORKS_PATH=botfinder
Another provider that is available is the API of the PEReN for which you need an api key
In order to impletement it you need to add to .env.local
two environment variables
BOT_SCORE_PROVIDER=peren
PEREN_API_KEY=XXXX
If you are part of AmbNum
, you can use the deploy scripts in the package.json
This may happen if you run yarn docker:run
on a M1
If this happen, well I did not find any solution except testing by building on another machine
Node v17.0.1
prefers IPv6 addresses over IPv4 but mongo by default is configured with ipv4.
You need to change mondodb.conf
configuration from
net:
bindIp: 127.0.0.1
to
net:
ipv6: true
port: 27017
bindIpAll: true
and restart mongodb-community brew services restart mongodb-community
Good to know : the mongodb configuration file is available in /usr/local/etc/mongod.conf
(macOS) or in /opt/homebrew/etc/mongod.conf
(M1 processor). For other OS, read the doc