tribler.core.database.store =========================== .. py:module:: tribler.core.database.store Attributes ---------- .. autoapisummary:: tribler.core.database.store.BETA_DB_VERSIONS tribler.core.database.store.CURRENT_DB_VERSION tribler.core.database.store.MIN_BATCH_SIZE tribler.core.database.store.MAX_BATCH_SIZE tribler.core.database.store.POPULAR_TORRENTS_FRESHNESS_PERIOD tribler.core.database.store.POPULAR_TORRENTS_COUNT tribler.core.database.store.sql_create_fts_table tribler.core.database.store.sql_add_fts_trigger_insert tribler.core.database.store.sql_add_fts_trigger_delete tribler.core.database.store.sql_add_fts_trigger_update tribler.core.database.store.sql_add_torrentstate_trigger_after_insert tribler.core.database.store.sql_add_torrentstate_trigger_after_update tribler.core.database.store.sql_create_partial_index_torrentstate_last_check Classes ------- .. autoapisummary:: tribler.core.database.store.ObjState tribler.core.database.store.ProcessingResult tribler.core.database.store.MetadataStore Module Contents --------------- .. py:class:: ObjState(*args, **kwds) Bases: :py:obj:`enum.Enum` Different states of information in the store. .. py:attribute:: UPDATED_LOCAL_VERSION .. py:attribute:: LOCAL_VERSION_NEWER .. py:attribute:: LOCAL_VERSION_SAME .. py:attribute:: NEW_OBJECT .. py:attribute:: DUPLICATE_OBJECT .. py:class:: ProcessingResult This class is used to return results of processing of a payload by process_payload. It includes a dictionary of the metadata (data), the database rowid, and the state of the object as indicated by the ObjState enum. .. py:attribute:: data :type: dict[str, Any] .. py:attribute:: obj_state :type: ObjState .. py:attribute:: rowid :type: list[int] .. py:data:: BETA_DB_VERSIONS :value: [0, 1, 2, 3, 4, 5] .. py:data:: CURRENT_DB_VERSION :value: 15 .. py:data:: MIN_BATCH_SIZE :value: 10 .. py:data:: MAX_BATCH_SIZE :value: 1000 .. py:data:: POPULAR_TORRENTS_FRESHNESS_PERIOD :value: 86400 .. py:data:: POPULAR_TORRENTS_COUNT :value: 100 .. py:data:: sql_create_fts_table :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ CREATE VIRTUAL TABLE IF NOT EXISTS FtsIndex USING FTS5 (title, content='ChannelNode', prefix = '2 3 4 5', tokenize='porter unicode61 remove_diacritics 1');""" .. raw:: html
.. py:data:: sql_add_fts_trigger_insert :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ CREATE TRIGGER IF NOT EXISTS fts_ai AFTER INSERT ON ChannelNode BEGIN INSERT INTO FtsIndex(rowid, title) VALUES (new.rowid, new.title); END;""" .. raw:: html
.. py:data:: sql_add_fts_trigger_delete :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ CREATE TRIGGER IF NOT EXISTS fts_ad AFTER DELETE ON ChannelNode BEGIN DELETE FROM FtsIndex WHERE rowid = old.rowid; END;""" .. raw:: html
.. py:data:: sql_add_fts_trigger_update :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ CREATE TRIGGER IF NOT EXISTS fts_au AFTER UPDATE ON ChannelNode BEGIN DELETE FROM FtsIndex WHERE rowid = old.rowid; INSERT INTO FtsIndex(rowid, title) VALUES (new.rowid, new.title); END;""" .. raw:: html
.. py:data:: sql_add_torrentstate_trigger_after_insert :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ CREATE TRIGGER IF NOT EXISTS torrentstate_ai AFTER INSERT ON TorrentState BEGIN UPDATE "TorrentState" SET has_data = (last_check > 0) WHERE rowid = new.rowid; END; """ .. raw:: html
.. py:data:: sql_add_torrentstate_trigger_after_update :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ CREATE TRIGGER IF NOT EXISTS torrentstate_au AFTER UPDATE ON TorrentState BEGIN UPDATE "TorrentState" SET has_data = (last_check > 0) WHERE rowid = new.rowid; END; """ .. raw:: html
.. py:data:: sql_create_partial_index_torrentstate_last_check :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ CREATE INDEX IF NOT EXISTS idx_torrentstate__last_check__partial ON TorrentState (last_check, seeders, leechers, self_checked) WHERE has_data = 1; """ .. raw:: html
.. py:class:: MetadataStore(db_filename: str, private_key: ipv8.keyvault.keys.PrivateKey, disable_sync: bool = False, notifier: tribler.core.notifier.Notifier | None = None, check_tables: bool = True, db_version: int = CURRENT_DB_VERSION) Storage of metadata for channels and torrents. .. py:attribute:: notifier :value: None .. py:attribute:: db_path .. py:attribute:: my_key .. py:attribute:: my_public_key_bin .. py:attribute:: _logger .. py:attribute:: _shutting_down :value: False .. py:attribute:: batch_size :value: 10 .. py:attribute:: reference_timedelta .. py:attribute:: sleep_on_external_thread :value: 0.05 .. py:attribute:: db .. py:attribute:: MiscData .. py:attribute:: TrackerState .. py:attribute:: TorrentState .. py:attribute:: TorrentMetadata .. py:method:: fast_integrity_check(remove_broken: bool = True) -> bool Inspect the database file and return whether it was broken or not. By default, this method also removes the given db file if it is broken. Set ``remove_broken`` to False if you want to keep the broken file. :returns: Whether the database file was broken (False if no errors occurred). .. py:method:: set_value(key: str, value: str) -> None Set a generic key to a value. .. py:method:: get_value(key: str, default: str | None = None) -> str | None Retrieve the value for a given key. .. py:method:: drop_indexes() -> None Drop the indices for this database. .. py:method:: get_objects_to_create() -> list[pony.orm.core.Entity] Get the objects that need to be created. .. py:method:: get_db_file_size() -> int Get the physical size on disk (always 0 for memory dbs). .. py:method:: drop_fts_triggers() -> None Drop the FTS triggers. .. py:method:: create_fts_triggers() -> None Create the FTS triggers. .. py:method:: fill_fts_index() -> None Insert the FTS indices. .. py:method:: create_torrentstate_triggers() -> None Create the torrent state triggers. .. py:method:: shutdown() -> None Disconnect the connection to the database. .. py:method:: run_threaded(func: collections.abc.Callable, *args: Any, **kwargs) -> Any :async: Run ``func`` threaded and close DB connection at the end of the execution. :param func: the function to be executed threaded :param args: args for the function call :param kwargs: kwargs for the function call :return: a result of the func call. .. py:method:: process_compressed_mdblob_threaded(compressed_data: bytes, **kwargs) -> list[ProcessingResult] :async: Decompress the given data in a thread and return a list of uncompressed results. .. py:method:: process_compressed_mdblob(compressed_data: bytes, skip_personal_metadata_payload: bool = True) -> list[ProcessingResult] Decompress the given data and return a list of uncompressed results. .. py:method:: process_torrent_health(health: tribler.core.torrent_checker.healthdataclasses.HealthInfo) -> bool Adds or updates information about a torrent health for the torrent with the specified infohash value. :param health: a health info of a torrent :return: True if a new TorrentState object was added .. py:method:: process_squashed_mdblob(chunk_data: bytes, external_thread: bool = False, health_info: list[tuple[int, int, int]] | None = None, skip_personal_metadata_payload: bool = True) -> list[ProcessingResult] Process raw concatenated payloads blob. This routine breaks the database access into smaller batches. It uses a congestion-control like algorithm to determine the optimal batch size, targeting the batch processing time value of self.reference_timedelta. :param chunk_data: the blob itself, consists of one or more GigaChannel payloads concatenated together :param external_thread: if this is set to True, we add some sleep between batches to allow other threads to get the database lock. This is an ugly workaround for Python and asynchronous programming (locking) imperfections. It only makes sense to use it when this routine runs on a non-reactor thread. :param health_info: the health info to update a torrent with. :param skip_personal_metadata_payload: don't process our own torrents. :return: a list of tuples of (, ) .. py:method:: process_payload(payload: tribler.core.database.serialization.TorrentMetadataPayload, skip_personal_metadata_payload: bool = True) -> list[ProcessingResult] Write a payload to our database (if necessary). .. py:method:: get_num_torrents() -> int Get the number of torrents in the database. .. py:method:: search_keyword(query: str, origin_id: int | None = None) -> pony.orm.core.Query Search for an FTS query, potentially restricted to a given origin id. Requires FTS5 table "FtsIndex" to be generated and populated. FTS table is maintained automatically by SQL triggers. BM25 ranking is embedded in FTS5. .. py:method:: get_entries_query(metadata_type: int | None = None, channel_pk: bytes | None = None, hide_xxx: bool = False, origin_id: int | None = None, sort_by: str | None = None, sort_desc: bool = True, max_rowid: int | None = None, txt_filter: str | None = None, category: str | None = None, infohash: bytes | None = None, infohash_set: set[bytes] | None = None, id_: int | None = None, self_checked_torrent: bool | None = None, health_checked_after: int | None = None, popular: bool | None = None, tags: list[str] | None = None, **kwargs) -> pony.orm.core.Query This method implements REST-friendly way to get entries from the database. Warning! For Pony magic to work, iteration variable name (e.g. 'g') should be the same everywhere! :return: PonyORM query object corresponding to the given params. .. py:method:: get_entries_threaded(**kwargs) -> list[tribler.core.database.orm_bindings.torrent_metadata.TorrentMetadata] :async: Retrieve entries in a thread and return a list of results. .. py:method:: get_entries(first: int = 1, last: int | None = None, **kwargs) -> list[tribler.core.database.orm_bindings.torrent_metadata.TorrentMetadata] Get some torrents. Optionally sort the results by a specific field, or filter the channels based on a keyword/whether you are subscribed to it. :return: A list of class members .. py:method:: get_total_count(**kwargs) -> int | None Get total count of torrents that would be returned if there would be no pagination/limits/sort. .. py:method:: get_entries_count(**kwargs) -> int | None Get the count of torrents that would be returned if there would be no pagination/limits. .. py:method:: get_max_rowid() -> int Get the highest-known row id. .. py:attribute:: fts_keyword_search_re .. py:method:: get_auto_complete_terms(text: str, max_terms: int) -> list[str] Get the auto-completion terms for a given query.