TypeError: can't compare offset-naive and offset-aware datetimes

https://mastodon.ml/@lina/112768409279788952

.gitea/workflows/gitea-actions.yaml

@@ -0,0 +1,25 @@
+name: Run tests
+run-name: ${{ gitea.actor }} is testing out test
+on: [push]
+
+jobs:
+  Run-Test:
+    runs-on: ubuntu-act-latest
+    steps:
+      - run: echo "🎉 The job was automatically triggered by a ${{ gitea.event_name }} event."
+      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by Gitea!"
+      - run: echo "🔎 The name of your branch is ${{ gitea.ref }} and your repository is ${{ gitea.repository }}."
+      - name: Check out repository code
+        uses: actions/checkout@v4
+      - run: echo "💡 The ${{ gitea.repository }} repository has been cloned to the runner."
+      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
+      - name: List files in the repository
+        run: |
+          ls ${{ gitea.workspace }}
+      - run: python3 -m venv venv
+      - run: . venv/bin/activate
+      - run: pip install -r requirements.txt
+      - run: pip install time_machine
+      - run: cp config.py.example config.py
+      - run: python test.py
+      - run: echo "🍏 This job's status is ${{ job.status }}."

.gitignore

@@ -4,4 +4,5 @@
 .app_sessions
 *.log
 *.sqlite
-
+*.json
+config.py

COPYING

@@ -0,0 +1,110 @@
+Creative Commons Legal Code CC0 1.0 Universal Official translations of this
+legal tool are available
+
+    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL
+    SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN ATTORNEY-CLIENT
+    RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS"
+    BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE USE OF THIS
+    DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER, AND DISCLAIMS
+    LIABILITY FOR DAMAGES RESULTING FROM THE USE OF THIS DOCUMENT OR THE
+    INFORMATION OR WORKS PROVIDED HEREUNDER. 
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator and
+subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for the
+purpose of contributing to a commons of creative, cultural and scientific works
+("Commons") that the public can reliably and without fear of later claims of
+infringement build upon, modify, incorporate in other works, reuse and
+redistribute as freely as possible in any form whatsoever and for any purposes,
+including without limitation commercial purposes. These owners may contribute
+to the Commons to promote the ideal of a free culture and the further
+production of creative, cultural and scientific works, or to gain reputation or
+greater distribution for their Work in part through the use and efforts of
+others.
+
+For these and/or other purposes and motivations, and without any expectation of
+additional consideration or compensation, the person associating CC0 with a
+Work (the "Affirmer"), to the extent that he or she is an owner of Copyright
+and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and
+publicly distribute the Work under its terms, with knowledge of his or her
+Copyright and Related Rights in the Work and the meaning and intended legal
+effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not limited to,
+the following:
+
+    the right to reproduce, adapt, distribute, perform, display, communicate,
+    and translate a Work; moral rights retained by the original author(s)
+    and/or performer(s); publicity and privacy rights pertaining to a person's
+    image or likeness depicted in a Work; rights protecting against unfair
+    competition in regards to a Work, subject to the limitations in paragraph
+    4(a), below; rights protecting the extraction, dissemination, use and reuse
+    of data in a Work; database rights (such as those arising under Directive
+    96/9/EC of the European Parliament and of the Council of 11 March 1996 on
+    the legal protection of databases, and under any national implementation
+    thereof, including any amended or successor version of such directive); and
+    other similar, equivalent or corresponding rights throughout the world
+    based on applicable law or treaty, and any national implementations
+    thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention of,
+applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and
+unconditionally waives, abandons, and surrenders all of Affirmer's Copyright
+and Related Rights and associated claims and causes of action, whether now
+known or unknown (including existing as well as future claims and causes of
+action), in the Work (i) in all territories worldwide, (ii) for the maximum
+duration provided by applicable law or treaty (including future time
+extensions), (iii) in any current or future medium and for any number of
+copies, and (iv) for any purpose whatsoever, including without limitation
+commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes
+the Waiver for the benefit of each member of the public at large and to the
+detriment of Affirmer's heirs and successors, fully intending that such Waiver
+shall not be subject to revocation, rescission, cancellation, termination, or
+any other legal or equitable action to disrupt the quiet enjoyment of the Work
+by the public as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason be
+judged legally invalid or ineffective under applicable law, then the Waiver
+shall be preserved to the maximum extent permitted taking into account
+Affirmer's express Statement of Purpose. In addition, to the extent the Waiver
+is so judged Affirmer hereby grants to each affected person a royalty-free, non
+transferable, non sublicensable, non exclusive, irrevocable and unconditional
+license to exercise Affirmer's Copyright and Related Rights in the Work (i) in
+all territories worldwide, (ii) for the maximum duration provided by applicable
+law or treaty (including future time extensions), (iii) in any current or
+future medium and for any number of copies, and (iv) for any purpose
+whatsoever, including without limitation commercial, advertising or promotional
+purposes (the "License"). The License shall be deemed effective as of the date
+CC0 was applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder of the
+License, and in such case Affirmer hereby affirms that he or she will not (i)
+exercise any of his or her remaining Copyright and Related Rights in the Work
+or (ii) assert any associated claims and causes of action with respect to the
+Work, in either case contrary to Affirmer's express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+    No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.  Affirmer
+    offers the Work as-is and makes no representations or warranties of any
+    kind concerning the Work, express, implied, statutory or otherwise,
+    including without limitation warranties of title, merchantability, fitness
+    for a particular purpose, non infringement, or the absence of latent or
+    other defects, accuracy, or the present or absence of errors, whether or
+    not discoverable, all to the greatest extent permissible under applicable
+    law.  Affirmer disclaims responsibility for clearing rights of other
+    persons that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work. Further,
+    Affirmer disclaims responsibility for obtaining any necessary consents,
+    permissions or other rights required for any use of the Work.  Affirmer
+    understands and acknowledges that Creative Commons is not a party to this
+    document and has no duty or obligation with respect to this CC0 or use of
+    the Work.

MIRRORS

@@ -0,0 +1,5 @@
+# Avalaible mirrors
+https://inex.dev/localhost_frssoft/FMN_bot.git
+https://git.macaw.me/localhost_frssoft/FMN_bot.git
+https://code.criminallycute.fi/localhost_frssoft/FMN_bot.git
+https://git.poridge.club/localhost_frssoft/FMN_bot.git

README.md

@@ -4,8 +4,10 @@
 Special for [XXIVProduction](https://xxivproduction.video/) and [drq@mastodon.ml](https://mastodon.ml/@drq).  
 
 ## Первичная инициализация
-* Установка зависимостей python3  
+* Установка зависимостей python3 и виртуального окружения  
 ```
+python3 -m venv venv
+. venv/bin/activate
 pip install -r requirements.txt
 ```  
 
@@ -33,7 +35,10 @@ browser=links ./auth_helper.sh
 Следуйте указаниям скрипта, залогиньтесь в ваш аккаунт и скопируйте код-ключ с браузера, закройте и вставьте в появившееся поле ввода.  
 
 * Настройка бота  
-В файле config.py описан каждый параметр, который можно менять  
+В файле config.py.example описан каждый параметр, который можно менять сделайте его копию в каталог с ботом убрав расширение .example 
+```
+cp config.py.example config.py
+```
 
 ## Запуск
 ```
@@ -41,15 +46,16 @@ python3 fmn_bot.py
 ```  
 
 ## Использование
-Просто упомяните бота (упоминающий должен быть прописан в администраторах бота см. config.py), в воскресенье/понедельник, когда Fediverse Movie Night окончен. Бот инициализирует сборщика предложений и будет собирать фильмы, которые будут поступать в виде ссылок на imdb.com (также теперь есть поддержка фронтедов [libremdb](https://github.com/zyachel/libremdb)) и kinopoisk.ru в том треде, где его упомянули. Сбор будет завершен во вторник 16:00 (по-умолчанию) MSK+3. После этого всех кто не успел предложить фильм бот будет уведомлять, что сбор завершен и даст ссылку на голосовалку. По завершению голосовалки (суббота 16:00), будет вычисляться фильм-победитель на FMN, если у нескольких фильмов одинаковые голоса, то будет создан tie breaker. Победивший фильм будет записан сразу как "просмотренный", чтобы не добавлять его на следующий FMN повторно. Далее бот будет ждать очередного упоминания.  
+Просто упомяните бота (упоминающий должен быть прописан в администраторах бота см. config.py), в воскресенье/понедельник, когда Fediverse Movie Night окончен (В новой версии, теперь встроена функция автоматического срабатывания в полночь, если упоминания не было). Бот инициализирует сборщика предложений и будет собирать фильмы, которые будут поступать в виде ссылок на imdb.com (также теперь есть поддержка фронтедов [libremdb](https://github.com/zyachel/libremdb)) и kinopoisk.ru в том треде, где его упомянули. Сбор будет завершен во вторник 16:00 (по-умолчанию) MSK+3. После этого всех кто не успел предложить фильм бот будет уведомлять, что сбор завершен и даст ссылку на голосовалку. По завершению голосовалки (суббота 16:00), будет вычисляться фильм-победитель на FMN, если у нескольких фильмов одинаковые голоса, то будет создан tie breaker. Победивший фильм будет записан сразу как "просмотренный", чтобы не добавлять его на следующий FMN повторно. Далее бот будет ждать очередного упоминания.  
 
 Note: Рекомендуется использовать ссылки на imdb.com, так как локальная база IMDB надёжнее, чем сетевой сторонний API Кинопоиска.    
 Note2: Список доступных для приёма инстансов libremdb обновляется вручную и может не соотвествовать официальному.  
   
 ## Список поддерживаемых инстансов libremdb:
-* https://libremdb.herokuapp.com
-* https://libremdb.pussthecat.org
-* https://libremdbeu.herokuapp.com	
+* https://libremdb.leemoon.network/ (Спасибо [Саре](https://lamp.leemoon.network/@sarahquartz) в рамках self-host проекта [Leemoon Network 🍋](https://leemoon.network))
+* https://libremdb.herokuapp.com/
+* https://libremdb.pussthecat.org/
+* https://libremdbeu.herokuapp.com/	
 * https://lmdb.tokhmi.xyz/
 * https://libremdb.esmailelbob.xyz/
 * http://libremdb.lqs5fjmajyp7rvp4qvyubwofzi6d4imua7vs237rkc4m5qogitqwrgyd.onion/  

config.py.example

@@ -1,7 +1,10 @@
 admins_bot = ('drq@mastodon.ml',) # Адреса админов бота, которые могут упомянуть бота для инициализации FMN
 # Example: ('admin_user', 'another_admin_user2@example.example') or ('admin_user',)
 bot_acct = 'fmn' # Ник бота на инстансе
-instance = 'expired.mentality.rip' # Инстанс, где будет запущен бот
+instance = 'pleroma.dark-alexandr.net' # Инстанс, где будет запущен бот
+# Ссылка на live stream FMN (API)
+peertube_stream_url = 'https://xxivproduction.video/api/v1/videos/1FZeVVVzWBFShaxQVkYiXd'
+
 
 # Лимиты
 limit_movies_per_user = 2 # Ограничение количества фильмов на одного пользователя
@@ -13,5 +16,5 @@ max_fail_limit = 4 # Игнорировать предложения польз
 hour_poll_posting = 16 # Час в который будет создан пост с голосовалкой (и завершение сбора)
 fmn_next_watching_hour = 21 # Час начала киносеанса
 
-logger_default_level = 'DEBUG' # Уровень логгирования TRACE, DEBUG, INFO, SUCCESS, WARNING, ERROR, CRITICAL 
+logger_default_level = 'INFO' # Уровень логгирования TRACE, DEBUG, INFO, SUCCESS, WARNING, ERROR, CRITICAL 
 

fmn_bot.py

@@ -1,20 +1,20 @@
-from src import listener_context, listener_mention, imdb_datasets_worker
+from src import listener_context, listener_mention
 from config import logger_default_level
 from loguru import logger
 import time
 import sys
 
 
-
 def main():
     logger.remove()
     logger.add(sink=sys.stderr, level=logger_default_level)
     logger.add(sink='suggestions.log', level='INFO', filter='src.listener_context')
 
-    listener_mention.run_scan_notif() # Слушаем упоминания в фоне
+    listener_mention.run_scan_notif()  # Слушаем упоминания в фоне
     time.sleep(1)
 
-    listener_context.scan_context_thread() # Слушаем тред на новые предложения фильмов
+    listener_context.scan_context_thread()  # Слушаем тред на новые предложения фильмов
+
 
 if __name__ == '__main__':
     main()

src/FMN.webp

@@ -0,0 +1 @@
+FMN_prev.webp

src/fedi_api.py

@@ -1,6 +1,5 @@
 from config import instance
 import time
-import json
 import requests
 from loguru import logger
 
@@ -9,9 +8,13 @@ instance_point = f"https://{instance}/api/v1"
 with open(".auth", mode='rt') as auth:
     tkn = auth.read().replace('\n', '')
 
-headers= {
-        "Authorization": "Bearer " + tkn
-}
+
+s = requests.Session()
+s.headers.update({
+        "Authorization": "Bearer " + tkn,
+        "Accept-encoding": 'gzip'
+})
+
 
 def get_notifications():
     params = {
@@ -21,7 +24,7 @@ def get_notifications():
     success = 0
     while success == 0:
         try:
-            r = requests.get(instance_point + "/notifications", json=params, headers=headers)
+            r = s.get(instance_point + "/notifications", json=params)
             r.raise_for_status()
             success = 1
             return r.json()
@@ -31,12 +34,11 @@ def get_notifications():
             logger.info('Retrying get notificatios...')
 
 
-
 def mark_as_read_notification(id_notification):
     success = 0
     while success == 0:
         try:
-            r = requests.post(instance_point + f"/notifications/{id_notification}/dismiss", headers=headers)
+            r = s.post(instance_point + f"/notifications/{id_notification}/dismiss")
             r.raise_for_status()
             success = 1
             return r.json()
@@ -47,24 +49,28 @@ def mark_as_read_notification(id_notification):
 
 
 def get_status_context(status_id):
+    retry = 0
     success = 0
     while success == 0:
         try:
-            r = requests.get(instance_point + f"/statuses/{status_id}/context", headers=headers)
+            r = s.get(instance_point + f"/statuses/{status_id}/context", timeout=30)
             r.raise_for_status()
             success = 1
             return r.json()
-        except:
+        except Exception as E:
             logger.exception(f'Ошибка получения контекста треда {status_id}')
             time.sleep(30)
             logger.info('Повторный запрос треда...')
-        
+            retry += 1
+            if retry > 5:
+                raise IOError(f'Фетчинг треда поломан! {E}')
+
 
 def get_status(status_id):
     success = 0
     while success == 0:
         try:
-            r = requests.get(instance_point + f"/statuses/{status_id}", headers=headers)
+            r = s.get(instance_point + f"/statuses/{status_id}")
             r.raise_for_status()
             success = 1
             return r.json()
@@ -74,8 +80,7 @@ def get_status(status_id):
             logger.info(f'Retrying get status {status_id}')
 
 
-
-def post_status(text, reply_to_status_id=None, poll_options=None, poll_expires=345600, attachments=None):
+def post_status(text, reply_to_status_id=None, poll_options=None, poll_expires=345600, attachments=None, visibility='unlisted'):
     poll = None
     if poll_options is not None:
         poll = {
@@ -86,7 +91,7 @@ def post_status(text, reply_to_status_id=None, poll_options=None, poll_expires=3
     params = {
         "status": text,
         "in_reply_to_id": reply_to_status_id,
-        "visibility": "unlisted",
+        "visibility": visibility,
         "content_type": "text/plain",
         "language": "ru",
         "poll": poll
@@ -96,7 +101,7 @@ def post_status(text, reply_to_status_id=None, poll_options=None, poll_expires=3
     success = 0
     while success == 0:
         try:
-            r = requests.post(instance_point + "/statuses", json=params, headers=headers)
+            r = s.post(instance_point + "/statuses", json=params)
             r.raise_for_status()
             success = 1
             return r.json()
@@ -112,19 +117,9 @@ def upload_attachment(file_path):
     params = {
         "description": "Fediverse Movie Night\nВоскресенье, 21:00\nLIVE ON XXIV Production",
     }
-    success = 0
-    while success == 0:
-        try:
-            r = requests.post(instance_point + "/media", params, files=file, headers=headers)
-            r.raise_for_status()
-            success = 1
-            return r.json()['id']
-        except:
-            logger.exception(f'Error uploading {file_path} attachment')
-            time.sleep(5)
-            logger.info(f'Retrying upload {file_path}...')
-
-
+    r = s.post(instance_point + "/media", params, files=file, timeout=30)
+    r.raise_for_status()
+    return r.json()['id']
 
 
 def mute_user(acct_id=str, acct=str, duration=None):
@@ -134,7 +129,7 @@ def mute_user(acct_id=str, acct=str, duration=None):
     success = 0
     while success == 0:
         try:
-            r = requests.post(instance_point + '/accounts' + f"/{acct_id}/mute", params, headers=headers)
+            r = s.post(instance_point + '/accounts' + f"/{acct_id}/mute", params)
             r.raise_for_status() 
             logger.info(f'Пользователь {acct} был заглушен на {duration} secs')
             success = 1
@@ -142,4 +137,3 @@ def mute_user(acct_id=str, acct=str, duration=None):
             logger.exception(f'Ошибка глушения {acct}')
             time.sleep(5)
             logger.info(f'Повторное глушение {acct}...')
-

src/fmn_database.py

@@ -13,6 +13,9 @@ c.execute(
 
 conn.commit()
 
+def force_commit():
+    conn.commit()
+
 
 def mark_as_watched_movie(original_name=None, ru_name=None, year=None):
     try:
@@ -36,6 +39,11 @@ def get_already_watched(original_name=None, ru_name=None, year=None):
     else:
         return False
 
+
+def get_count_all_watched_movies():
+    return len(c.execute('SELECT * FROM watched_movies').fetchall())
+
+
 def get_already_suggested(acct, original_name, ru_name, year):
     already_suggested = c.execute('''SELECT * FROM poll 
     WHERE (original_name = (?) OR ru_name = (?)) COLLATE NOCASE
@@ -105,4 +113,3 @@ def reset_poll():
     '''Сброс содержимого предложки-опроса'''
     c.execute("DELETE FROM poll")
     conn.commit()
-

src/fmn_poll.py

@@ -1,18 +1,24 @@
 from src.fedi_api import get_status, post_status, upload_attachment
-from src.fmn_states_db import add_state, get_state, clear_all_states
-from src.fmn_database import get_movies_for_poll, write_votes, read_votes, mark_as_watched_movie, get_already_watched, rewrite_db, reset_poll
+from src.fmn_states_db import states_stor, write_states
+from src.fmn_database import get_movies_for_poll, write_votes, read_votes, mark_as_watched_movie, get_already_watched, rewrite_db, reset_poll, get_count_all_watched_movies, force_commit
 from collections import Counter
 from loguru import logger
 import time
 
 
-text_create_poll = '''Друзья, голосование за следующий Fediverse Movie Night объявляю открытым!
+def text_create_poll():
+    force_commit()
+    count_poll = get_count_all_watched_movies()
+    text_poll = f'''Друзья, {count_poll} голосование за следующий Fediverse Movie Night объявляю открытым!
 
-Ставки сделаны, ставок больше нет
-'''.replace('\t', '')
+    Ставки сделаны, ставок больше нет
+    '''.replace('\t', '')
+    return text_poll
 
 
-def create_poll_movies(text=text_create_poll, poll_expires=345600):
+@logger.catch
+def create_poll_movies(text=text_create_poll(), poll_expires=345600):
+    logger.debug('Creating poll')
     formated_poll_options = []
     raw_poll = get_movies_for_poll()
     for i in raw_poll:
@@ -21,6 +27,7 @@ def create_poll_movies(text=text_create_poll, poll_expires=345600):
         ru_name = i[2]
         year = i[3]
         poll_option_string = f"{ru_name} / {orig_name}, {year} ({acct})"
+        logger.debug(f"Adding option in poll: {poll_option_string}")
         if ru_name is None:
             poll_option_string = f"{orig_name}, {year} ({acct})"
         if orig_name is None:
@@ -28,16 +35,26 @@ def create_poll_movies(text=text_create_poll, poll_expires=345600):
         if len(poll_option_string) >= 200:
             poll_option_string = poll_option_string[0:199] # Обрезка на 200 символов.
         formated_poll_options.append(poll_option_string)
+
+    attaches = []
+    try:
+        attaches = [upload_attachment('src/FMN.webp')]
+    except Exception as E:
+        logger.error(f"attachements can't do upload: {E}")
+
     poll_status_id = post_status(text, None, formated_poll_options,
-            poll_expires=poll_expires, attachments=[upload_attachment('src/FMN.png')])
+            poll_expires=poll_expires, attachments=attaches)
     logger.success('Голосовалка создана')
-    add_state('poll_expires_at', int(time.time()) + poll_expires)
-    add_state('poll_status_id', poll_status_id['id'])
+    states_stor.states['poll_expires_at'] = int(time.time()) + poll_expires
+    states_stor.states['poll_status_id'] = poll_status_id['id']
+    write_states(states_stor.states)
     return poll_status_id
 
 
+@logger.catch
 def get_winner_movie(poll_status_id=str):
     '''Отмечаем победивший фильм на голосовании как просмотренный или постим tie breaker'''
+    states = states_stor.states
     votes_counters = []
     status_with_poll = get_status(poll_status_id)
     poll = status_with_poll['poll']
@@ -45,7 +62,7 @@ def get_winner_movie(poll_status_id=str):
     for option in poll['options']:
         votes_count = option['votes_count']
         votes_counters.append(votes_count)
-    
+
     write_votes(votes_counters)
     voted_movies = read_votes()
     max_vote = voted_movies[0][4]
@@ -57,7 +74,7 @@ def get_winner_movie(poll_status_id=str):
     if len(winned_movies) > 1:
         logger.warning('Будет создан tie breaker')
         rewrite_db(winned_movies)
-        if get_state('tie_breaker'):
+        if states.get('tie_breaker'):
             create_tie_breaker(2)
         else:
             create_tie_breaker()
@@ -71,20 +88,23 @@ def get_winner_movie(poll_status_id=str):
             win_variant = f"{orig_name}, {year}"
         if orig_name is None:
             win_variant = f"{ru_name}, {year}"
-        text_winned = f"Голосование завершилось! Победил вариант предложенный @{acct_suggested}:\n{win_variant}"
+        expired_poll_count = get_count_all_watched_movies() - 1
+        text_winned = f"{expired_poll_count} голосование завершилось! Победил вариант предложенный @{acct_suggested}:\n{win_variant}"
 
         logger.success("Победил " + str(movie))
-        post_status(text_winned, attachments=[upload_attachment('src/FMN.png')])
-        clear_all_states()
+        post_status(text_winned, attachments=[upload_attachment('src/FMN.webp')])
+        states_stor.states = {}
+        write_states()
         reset_poll()
 
 
+@logger.catch
 def create_tie_breaker(count_tie=1):
     '''Создание tie breaker'''
     if count_tie == 1:
-        add_state('tie_breaker', 1)
+        states_stor.states['tie_breaker'] = 1
+        write_states(states_stor.states)
         poll_expires = 8*60*60
     else:
         poll_expires = 4*60*60
     tie_poll = create_poll_movies("TIE BREAKER!!!\n\nВыбираем из победителей!", poll_expires)
-

src/fmn_states_db.py

@@ -1,41 +1,33 @@
-import sqlite3
+import json
 from loguru import logger
 
-
-conn = sqlite3.connect("fmn_states.sqlite", check_same_thread=False)
-c = conn.cursor()
-
-c.execute(
-    '''CREATE TABLE IF NOT EXISTS states(key VARCHAR (500) UNIQUE DEFAULT NULL, value VARCHAR (500) UNIQUE DEFAULT NULL)''')
-
-conn.commit()
+states_file = 'fmn_states.json'
 
 
-def add_state(key, value):
-    '''Создание стейта, если существует - будет заменен'''
-    logger.debug(f'Adding {key}: {value}')
-    c.execute("INSERT OR REPLACE INTO states(key, value) VALUES (?, ?)", (key, value))
-    conn.commit()
-    logger.warning(f'Добавлен стейт {key}: {value}')
+class states_stor:
+    states = None
 
 
-def get_state(key):
-    '''Получение стейта по ключу'''
-    logger.trace(f'Запрошен стейт {key}')
-    value = c.execute("SELECT value FROM states WHERE key = (?)", (key,)).fetchone()
-    if value:
-        return value[0]
+@logger.catch
+def read_states():
+    try:
+        with open(states_file, 'rt') as f:
+            current_states = json.loads(f.read())
+    except:
+        logger.warning('Стейты не найдены, создание плейсхолдера')
+        write_states()
+        current_states = {}
+    return current_states
 
 
-def remove_state(key):
-    '''Удалить стейт по ключу'''
-    c.execute("DELETE FROM states WHERE key = (?)", (key,))
-    conn.commit()
-    logger.warning(f'Удален стейт {key}')
+@logger.catch
+def write_states(new_states={}):
+    with open(states_file, 'wt') as f:
+        f.write(json.dumps(new_states, indent=4))
+    if new_states == {}:
+        logger.info('states empty wrote')
+    return new_states
 
 
-def clear_all_states():
-    c.execute("DELETE FROM states")
-    conn.commit()
-    logger.warning(f'Все стейты удалены')
-
+if not states_stor.states:
+    states_stor.states = read_states()

src/imdb_datasets_worker.py

@@ -1,5 +1,6 @@
 import sqlite3
 import gzip
+from loguru import logger
 
 conn = sqlite3.connect("imdb_titles.sqlite")
 c = conn.cursor()
@@ -26,15 +27,20 @@ def convert_tsv_to_db(title_basics_tsv):
                 original_name = line[3]
                 ru_name = None
                 year = line[5]
+                genres = line[-1].strip().split(',')
                 if year.startswith(r"\N"):
                     year = None
                 else:
                     year = int(year)
                     
-                if tt_type not in ("movie", "video"):
+                if tt_type not in ("movie", "tvMovie", "video"):
                     original_name = None
                     year = None
-
+                if "Documentary" in genres:
+                    logger.debug(f'Документальный {original_name} отсеян')
+                    original_name = None
+                    year = None
+                    tt_type = "doc"
                 write_dataset.append((tt_id, tt_type, original_name, ru_name, year))
                 counter += 1
                 if counter >= chunk:
@@ -42,9 +48,9 @@ def convert_tsv_to_db(title_basics_tsv):
                     write_dataset = []
                     counter = 0
                     progress_counter += chunk
-                    print(f'Обработано: {progress_counter}')
-            except Exception as E:
-                print(E)
+                    logger.info(f'Обработано: {progress_counter}')
+            except:
+                logger.exception('Err')
                 pass
         conn.commit()
 
@@ -61,16 +67,16 @@ def extract_ru_locale_from_tsv(title_akas_tsv):
                     continue
 
                 tt_id = int(line[0].split("tt")[1])
-                tt_type = c.execute(f"SELECT type FROM titles WHERE tt_id={tt_id}").fetchone()[0]
-                if tt_type not in ("movie", "video"):
+                tt_type = c.execute("SELECT type FROM titles WHERE tt_id = (?)", (tt_id, )).fetchone()[0]
+                if tt_type not in ("movie", "tvMovie", "video"):
                     continue
                 ru_name = line[2]
                 ru_name_writer.append((ru_name, tt_id))
                 counter += 1
-                print(f'Обработано ru_name: {counter}')
+                logger.info(f'Обработано ru_name: {counter}')
 
-            except Exception as E:
-                print(E)
+            except:
+                logger.exception('Err')
                 pass
         
         c.executemany("UPDATE titles SET ru_name = ? WHERE tt_id = ?", ru_name_writer)

src/listener_context.py

@@ -3,13 +3,10 @@ from src.fedi_api import get_status_context, get_status, post_status, mute_user
 from src.kinopoisk_api import get_kinopoisk_movie_to_imdb
 from src.imdb_datasets_worker import get_title_by_id
 from src.fmn_database import add_movie_to_poll, get_already_watched, get_suggested_movies_count
-from src.fmn_states_db import get_state, add_state
+from src.fmn_states_db import states_stor
 from src.fmn_poll import create_poll_movies, get_winner_movie
 import re
 import time
-from datetime import datetime
-from dateutil.parser import parse as dateutilparse
-from dateutil.relativedelta import relativedelta, TU
 from collections import Counter
 from loguru import logger
 
@@ -23,38 +20,48 @@ def parse_links(text=str):
 
 
 def parse_links_imdb(text=str):
-    regex = r"imdb\.com/|libremdb\.pussthecat\.org/|libremdb\.esmailelbob\.xyz/|libremdb\.herokuapp\.com/|libremdbeu\.herokuapp\.com/|lmdb\.tokhmi\.xyz/|libremdb\.lqs5fjmajyp7rvp4qvyubwofzi6d4imua7vs237rkc4m5qogitqwrgyd\.onion/"
+    regex = r"imdb\.com/|libremdb\.leemoon\.network/|libremdb\.pussthecat\.org/|libremdb\.esmailelbob\.xyz/|libremdb\.herokuapp\.com/|libremdbeu\.herokuapp\.com/|lmdb\.tokhmi\.xyz/|libremdb\.lqs5fjmajyp7rvp4qvyubwofzi6d4imua7vs237rkc4m5qogitqwrgyd\.onion/"
     if re.search(regex, text.lower(), flags=re.MULTILINE):
         imdb_ids = re.findall(r"tt(\d{1,})", text.lower())
         if imdb_ids != []:
             return imdb_ids[:limit_movies_per_user]
 
-        
+
 def scan_context_thread():
     fail_limit = Counter()
     while True:
-        status_id = get_state('last_thread_id')
-        poll_created = get_state('poll_status_id')
-        stop_thread_scan = get_state('stop_thread_scan')
+        states = states_stor.states
+        status_id = states.get('last_thread_id')
+        poll_created = states.get('poll_status_id')
+        stop_thread_scan = states.get('stop_thread_scan')
         time_now = int(time.time())
+        reserve_time = False
         while status_id is None or stop_thread_scan is None:
+            states = states_stor.states
             fail_limit = Counter()
-            status_id = get_state('last_thread_id')
-            stop_thread_scan = get_state('stop_thread_scan')
+            status_id = states.get('last_thread_id')
+            stop_thread_scan = states.get('stop_thread_scan')
             time.sleep(1)
 
         if time_now >= int(stop_thread_scan):
+            reserve_time = True
             logger.debug('Сбор завершён, сканирование треда на опоздавших')
             if poll_created is None:
                 create_poll_movies()
-                poll_created = get_state('poll_status_id')
+                poll_created = states_stor.states.get('poll_status_id')
             else:
-                if time_now >= int(get_state('poll_expires_at')):
+                if time_now >= int(states.get('poll_expires_at')):
                     get_winner_movie(poll_created)
         else:
             endings = int(stop_thread_scan) - time_now
             logger.debug(f'Осталось до закрытия сбора: {endings}')
+        if reserve_time:  # Reduce instance load
+            time.sleep(30)
+        get_thread_time = time.time()
         descendants = get_status_context(status_id)['descendants']
+        get_thread_time2 = time.time()
+        get_thread_delta = get_thread_time2 - get_thread_time
+        logger.debug(f'Get thread time: {get_thread_delta}')
         replyed = []
         for status in descendants:
             if status['account']['acct'] == bot_acct:
@@ -62,6 +69,10 @@ def scan_context_thread():
 
         for status in descendants:
             id_st = status['id']
+            visibility = status['visibility']
+            if visibility == 'direct':
+                # Игнорируем личку
+                continue
             in_reply_acct = status['in_reply_to_account_id']
             in_reply_id = status['in_reply_to_id']
             muted = status['muted']
@@ -69,14 +80,14 @@ def scan_context_thread():
             acct_id = status['account']['id']
             content = status['pleroma']['content']['text/plain']
 
-            if id_st in replyed: # Игнорировать уже отвеченное
+            if id_st in replyed:  # Игнорировать уже отвеченное
                 continue
             if muted is True:
                 continue
-            if fail_limit[acct] >= max_fail_limit: # Игнорировать пользователя если он превысил fail limit
-                mute_user(acct_id, acct, int(get_state('max_mute_time')) - time_now)
+            if fail_limit[acct] >= max_fail_limit:  # Игнорировать пользователя если он превысил fail limit
+                mute_user(acct_id, acct, int(states.get('max_mute_time')) - time_now)
                 logger.warning(f'{acct} игнорируется - превышение fail limit')
-                break # Нужно обновить тред, чтобы muted на заглушенном стал True
+                break  # Нужно обновить тред, чтобы muted на заглушенном стал True
 
             parsed_result = parse_links(content)
             parsed_result_imdb = parse_links_imdb(content)
@@ -89,29 +100,40 @@ def scan_context_thread():
                 logger.info(f'{acct} был уведомлен о завершенной голосовалке')
                 fail_limit[acct] += 1
                 continue
-            
+
             index_type = 1
             index_name = 2
             index_ru_name = 3
             index_year = 4
+
+            message_writer = []
+            success = False
+            if parsed_result and parsed_result_imdb:
+                post_status('❌ Не смешивайте IMDB и кинопоиск в одном посте, пожалуйста.', id_st)
+                fail_limit[acct] += 1
+                continue
             if parsed_result is not None:
                 print(parsed_result)
                 suggested_movies = get_kinopoisk_movie_to_imdb(parsed_result)
+                message_writer.append('⚠️  внимание при использовании Кинопоиска стабильность может быть понижена')
                 if suggested_movies is None:
-                    post_status('❌ Не удалось выполнить запрос: возможно некорректный тип фильма, попробуйте использовать imdb.com', id_st)
+                    post_status('❌ Не удалось выполнить запрос: возможно некорректный тип фильма, попробуйте использовать https://libremdb.leemoon.network', id_st)
                     fail_limit[acct] += 1
                     continue
             elif parsed_result_imdb is not None:
                 suggested_movies = get_title_by_id(parsed_result_imdb)
-           
-            message_writer = []
-            success = False
+
+                if suggested_movies is None:
+                    post_status('❌ Фильм(ы) не найден в базе данных IMDB, пожалуйста обратитесь к администратору, чтобы обновить базу. Примечание: IMDB выкладывает новые изменения не сразу.', id_st)
+                    fail_limit[acct] += 1
+                    continue
+
             for movie in suggested_movies:
                 logger.debug(str(movie))
                 if movie[index_type] == "404":
                     message_writer.append("❌ Не найдено.")
                     fail_limit[acct] += 1
-                elif movie[index_type] not in ("movie", "video"):
+                elif movie[index_type] not in ("movie", "tvMovie", "video"):
                     type_of_title = movie[index_type]
                     message_writer.append(f"❌ Не принято:\n- Нам не подходят: сериалы, короткометражные и документальные фильмы")
                     logger.info(f'Предложение {acct} отклонено: не подходящий тип фильма: {type_of_title}')
@@ -122,7 +144,7 @@ def scan_context_thread():
                     name_ru = movie[index_ru_name]
                     year = movie[index_year]
                     movie_string = f"{name_ru} / {name}, {year}"
-                    
+
                     if name is None:
                         movie_string = f"{name_ru}, {year}"
                     if name_ru is None:
@@ -136,8 +158,8 @@ def scan_context_thread():
                         logger.warning(f'Предложение {acct} было отклонено: количество уже предложенных фильмов превышает\равно {limit_all_movies_poll}')
                         fail_limit[acct] += 1
                         break
-                        
-                    if get_already_watched(name, name_ru, year) == True:
+
+                    if get_already_watched(name, name_ru, year) is True:
                         message_writer.append(f"ℹ️ Этот фильм уже был на FMN: {movie_string}")
                         logger.info(f'Попытка предложить уже просмотренный фильм: {acct} {name} {name_ru} {year}')
                         fail_limit[acct] += 1
@@ -153,7 +175,7 @@ def scan_context_thread():
                         logger.info(f'Предложение от {acct} было отлонено - фильм в опросе существует')
                         fail_limit[acct] += 1
                     else:
-                        message_writer.append("❌ Вы не можете добавить больше 2х фильмов")
+                        message_writer.append(f"❌ Вы не можете добавить больше {limit_movies_per_user}x фильмов")
                         logger.info(f'Предложение от {acct} было отлонено - лимит на пользователя')
                         fail_limit[acct] += 1
             if message_writer != []:
@@ -163,5 +185,3 @@ def scan_context_thread():
                 post_status('\n'.join(message_writer) + message, id_st)
 
         time.sleep(30)
-
-

src/listener_mention.py

@@ -1,61 +1,128 @@
 from src.fedi_api import get_notifications, mark_as_read_notification, post_status, upload_attachment
-from src.fmn_states_db import add_state, get_state
-from config import admins_bot, limit_movies_per_user, limit_all_movies_poll, hour_poll_posting, fmn_next_watching_hour
+from src.fmn_states_db import write_states, states_stor
+from src.sheduler import check_stop_thread_scan
+from config import admins_bot, limit_movies_per_user, limit_all_movies_poll, hour_poll_posting, fmn_next_watching_hour, peertube_stream_url
 
-import threading, time
+import threading
+import time
+import requests
 from datetime import datetime
 from dateutil.parser import parse as dateutilparse
 from dateutil.relativedelta import relativedelta, TU, SU
 from loguru import logger
 
 
+@logger.catch
+def get_peertube_stream_name():
+    try:
+        return requests.get(peertube_stream_url).json()['name']
+    except Exception as E:
+        return f"[не удалось получить название {E}]"
+
+
+@logger.catch
 def get_control_mention():
     while True:
+        states = states_stor.states
         time.sleep(30)
         time_now = datetime.now()
         now_week = time_now.weekday()
         now_hour = time_now.hour
         if now_week not in (0, 6):
             continue
-        if now_week == 6 and now_hour < fmn_next_watching_hour: # Предотвращение работы в холстую до начала сеанса
+        if now_week == 6 and now_hour < fmn_next_watching_hour:
+            # Предотвращение работы в холстую до начала сеанса
             continue
-        post_exists = get_state('last_thread_id')
+        post_exists = states.get('last_thread_id')
         if post_exists:
             continue
         logger.debug('Wait for from admin mention...')
         notif = get_notifications()
         for i in notif:
-            if i['type'] != "mention":
+            if i['type'] not in ("mention"):
                 continue
             seen = i['pleroma']['is_seen']
             acct_mention = i['account']['acct']
             reply_to_id = i['status']['in_reply_to_id']
-            if acct_mention in admins_bot and seen == False and reply_to_id == None and now_week in (0, 6):
+            if acct_mention in admins_bot and seen is False and reply_to_id is None and now_week in (0, 6):
                 logger.success(f'Найдено упоминание от {acct_mention}')
                 st_id = i['status']['id']
                 st_date = i['status']['created_at']
                 thread_created_at = dateutilparse(st_date)
 
-                delta = relativedelta(hour=hour_poll_posting, minute=0, second=0, weekday=TU(1))
+                delta = relativedelta(
+                    hour=hour_poll_posting, minute=0, second=0, weekday=TU(1))
                 stop_thread_scan = thread_created_at + delta
-                movies_accept_time = stop_thread_scan.strftime('%H:%M %d.%m.%Y по Москве')
-                stop_thread_scan = time.mktime(time.struct_time(stop_thread_scan.timetuple()))
-                
-                if now_week == 6: # Фикс стыков двух недель. Если вс, то расчитываем на следующую неделю
+                # if check_stop_thread_scan(stop_thread_scan) == stop_thread_scan: # broken because pytz
+                #    logger.success("проверка на корректность даты пройдена")
+                # else:
+                #   logger.warning("проверка на корректность даты не пройдена")
+                    # stop_thread_scan = check_stop_thread_scan(stop_thread_scan)
+                movies_accept_time = stop_thread_scan.strftime(
+                    '%H:%M %d.%m.%Y по Москве')
+                stop_thread_scan = time.mktime(
+                    time.struct_time(stop_thread_scan.timetuple()))
+
+                if now_week == 6:  # Фикс стыков двух недель. Если вс, то расчитываем на следующую неделю
                     next_week = 2
                 else:
                     next_week = 1
-                next_movie_watching_delta = relativedelta(hour=fmn_next_watching_hour, minute=0, second=0, weekday=SU(next_week))
+                next_movie_watching_delta = relativedelta(
+                    hour=fmn_next_watching_hour, minute=0, second=0, weekday=SU(next_week))
                 next_movie_watching = time_now + next_movie_watching_delta
-                max_mute_time = time.mktime(time.struct_time(next_movie_watching.timetuple())) # Глушение до следующего сеанса FMN.
+                # Глушение до следующего сеанса FMN.
+                max_mute_time = time.mktime(
+                    time.struct_time(next_movie_watching.timetuple()))
                 next_movie_watching = next_movie_watching.strftime('%d.%m.%Y')
-                post_status(start_collect_movies_text(movies_accept_time, next_movie_watching), st_id, attachments=[upload_attachment('src/FMN.png')])
+                post_status(start_collect_movies_text(movies_accept_time, next_movie_watching),
+                            st_id, attachments=[upload_attachment('src/FMN.webp')])
 
                 time.sleep(0.2)
                 mark_as_read_notification(i['id'])
-                add_state('max_mute_time', int(max_mute_time))
-                add_state('stop_thread_scan', int(stop_thread_scan))
-                add_state('last_thread_id', st_id)
+
+                states_stor.states['max_mute_time'] = int(max_mute_time)
+                states_stor.states['stop_thread_scan'] = int(stop_thread_scan)
+                states_stor.states['last_thread_id'] = st_id
+                write_states(states_stor.states)
+                break
+            if now_hour == 0:
+                logger.warning('Автоматический триггер в полночи сработал')
+                thread_created_at = time_now
+
+                delta = relativedelta(
+                    hour=hour_poll_posting, minute=0, second=0, weekday=TU(1))
+                stop_thread_scan = thread_created_at + delta
+                if check_stop_thread_scan(stop_thread_scan) == stop_thread_scan:
+                    logger.success("проверка на корректность даты пройдена")
+                else:
+                    logger.warning("проверка на корректность даты не пройдена")
+                    # stop_thread_scan = check_stop_thread_scan(stop_thread_scan)
+                movies_accept_time = stop_thread_scan.strftime(
+                    '%H:%M %d.%m.%Y по Москве')
+                stop_thread_scan = time.mktime(
+                    time.struct_time(stop_thread_scan.timetuple()))
+
+                if now_week == 6:  # Фикс стыков двух недель. Если вс, то расчитываем на следующую неделю
+                    next_week = 2
+                else:
+                    next_week = 1
+                next_movie_watching_delta = relativedelta(
+                    hour=fmn_next_watching_hour, minute=0, second=0, weekday=SU(next_week))
+                next_movie_watching = time_now + next_movie_watching_delta
+                # Глушение до следующего сеанса FMN.
+                max_mute_time = time.mktime(
+                    time.struct_time(next_movie_watching.timetuple()))
+                next_movie_watching = next_movie_watching.strftime('%d.%m.%Y')
+                watched_movie_name = get_peertube_stream_name()
+                st_id = post_status("Спасибо что посмотрели " + watched_movie_name + "\n\n" + start_collect_movies_text(movies_accept_time, next_movie_watching) +
+                                    '\n\n@rf@mastodon.ml', reply_to_status_id=None, attachments=[upload_attachment('src/FMN.webp')], visibility="public")['id']
+
+                time.sleep(0.2)
+
+                states_stor.states['max_mute_time'] = int(max_mute_time)
+                states_stor.states['stop_thread_scan'] = int(stop_thread_scan)
+                states_stor.states['last_thread_id'] = st_id
+                write_states(states_stor.states)
                 break
         time.sleep(30)
 
@@ -66,19 +133,20 @@ def start_collect_movies_text(movies_accept_time=str, next_movie_watching=str):
 
 Напоминаем правила:
  - Мы принимаем на просмотр полнометражные художественные фильмы;
- - Прием варианта осуществляется путем публикации ссылки на этот фильм на IMDB (libremdb) или Кинопоиске в этом треде;
+ - Прием варианта осуществляется путем публикации ссылки на этот фильм на IMDB https://libremdb.leemoon.network/ или Кинопоиске в этом треде;
  - Нам не подходят: сериалы, короткометражные и документальные фильмы;
  - Максимальное количество вариантов, предложенных одним человеком не должно превышать {limit_movies_per_user};
  - Всего может быть собрано до {limit_all_movies_poll} фильмов;
  - Заявки принимаются до крайнего срока, после чего будет объявлено голосование по собранным вариантам.
 
 Крайний срок подачи заявки - {movies_accept_time}.
+Рекомендуем посетить список, чтобы случайно не предложить уже просмотренный фильм: https://pub.phreedom.club/~localhost/fmn_watched.gmi
 
 Желаем удачи.
     '''.replace('\t', '')
     return text
 
+
 def run_scan_notif():
     scan_notif = threading.Thread(target=get_control_mention, daemon=True)
     scan_notif.start()
-

src/sheduler.py

@@ -0,0 +1,23 @@
+import calendar
+import datetime
+from config import hour_poll_posting
+
+
+def check_stop_thread_scan(suggested_date):
+    suggested_month = suggested_date.month
+    suggested_year = suggested_date.year
+    tuesdays = []
+    for i in calendar.Calendar().itermonthdays4(suggested_year, suggested_month):
+        if i[3] == 1 and datetime.datetime(year=i[0], month=i[1], day=i[2]) > suggested_date:
+            tuesdays.append(datetime.datetime(
+                year=i[0], month=i[1], day=i[2], hour=hour_poll_posting))
+    if tuesdays == []:
+        shift_for_next_week = suggested_date + datetime.timedelta(days=1)
+        suggested_month = shift_for_next_week.month
+        suggested_year = shift_for_next_week.year
+        for i in calendar.Calendar().itermonthdays4(suggested_year, suggested_month):
+            if i[3] == 1 and datetime.datetime(year=i[0], month=i[1], day=i[2]) > suggested_date:
+                tuesdays.append(datetime.datetime(
+                    year=i[0], month=i[1], day=i[2], hour=hour_poll_posting))
+
+    return tuesdays[0]  # near tuesday

test.py

@@ -0,0 +1,30 @@
+import unittest
+import datetime
+import time_machine
+from src import sheduler
+
+
+class TestDate(unittest.TestCase):
+    def test_date_tue(self):
+        """
+        test near tue
+        тестирует корректность расчета даты на след. вторник
+        условия: расчет начат от воскресенья или понедельника (случайно)
+        """
+        import calendar
+        import random
+        for month in range(1, 13):
+            sundays = [i for i in calendar.Calendar().itermonthdays4(year=2024, month=month) if i[3] == 6]
+            for sunday in sundays:
+                fake_date = datetime.datetime(year=sunday[0], month=sunday[1], day=sunday[2], hour=random.randint(21, 23)) + datetime.timedelta(days=random.randint(0, 1))
+                with time_machine.travel(fake_date):
+                    try:
+                        result = sheduler.check_stop_thread_scan(fake_date)
+                        print("current fake date: " + fake_date.strftime("%Y.%m.%d %H:%M") + f" near tue: {result}")
+                    except Exception as E:
+                        print("fail fake date: " + fake_date.strftime("%Y.%m.%d %H:%M") + f", err: {E}")
+
+
+
+if __name__ == '__main__':
+    unittest.main()