> For the complete documentation index, see [llms.txt](https://lijeffrey39.gitbook.io/sentiment-ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lijeffrey39.gitbook.io/sentiment-ai/project-setup/tweet-setup.md).

# Stock tweets

To begin backtesting, stock tweets per day must be stored locally given a range of dates. While there are around 5000 stocks being monitored and scraped from stocktwits, many of these stocks don't receive a significant number of tweets on a daily basis. Therefore, for each trading day, only the top 100 most tweeted about stocks are saved for backtesting. After determining the daily top stocks, the tweets must be queried for by date. An example query here finds all bullish/bearish AAPL tweets between March 1st and March 2nd.

```python
{
  {'symbol': 'AAPL'},
  {"$or": [
    {'isBull': True},
    {'isBull': False}
  ]},
  {'time': 
    {
      '$gte': datetime.datetime(2019, 3, 1),
      '$lt': datetime.datetime(2019, 3, 2)
    }
  }
}
```

Once fetched, each stock's tweets can be efficiently accessed based on the following example data structure.

```python
{
    '2019-03-01': [
        [True, 'user1', '2019-03-01 13:40:00'],
        [True, 'user2', '2019-03-01 14:42:00'],
        [False, 'user3', '2019-03-01 14:53:00'],
        ...
    ],
    ...
}
```

The data for each stock is then stored in the `stock_files` folder as pickle files (ex. `stock_files/AAPL.pkl`).