tribler.core.database.store
===========================
.. py:module:: tribler.core.database.store
Attributes
----------
.. autoapisummary::
tribler.core.database.store.BETA_DB_VERSIONS
tribler.core.database.store.CURRENT_DB_VERSION
tribler.core.database.store.MIN_BATCH_SIZE
tribler.core.database.store.MAX_BATCH_SIZE
tribler.core.database.store.POPULAR_TORRENTS_FRESHNESS_PERIOD
tribler.core.database.store.POPULAR_TORRENTS_COUNT
tribler.core.database.store.sql_create_fts_table
tribler.core.database.store.sql_add_fts_trigger_insert
tribler.core.database.store.sql_add_fts_trigger_delete
tribler.core.database.store.sql_add_fts_trigger_update
tribler.core.database.store.sql_add_torrentstate_trigger_after_insert
tribler.core.database.store.sql_add_torrentstate_trigger_after_update
tribler.core.database.store.sql_create_partial_index_torrentstate_last_check
Classes
-------
.. autoapisummary::
tribler.core.database.store.ObjState
tribler.core.database.store.ProcessingResult
tribler.core.database.store.MetadataStore
Module Contents
---------------
.. py:class:: ObjState(*args, **kwds)
Bases: :py:obj:`enum.Enum`
Different states of information in the store.
.. py:attribute:: UPDATED_LOCAL_VERSION
.. py:attribute:: LOCAL_VERSION_NEWER
.. py:attribute:: LOCAL_VERSION_SAME
.. py:attribute:: NEW_OBJECT
.. py:attribute:: DUPLICATE_OBJECT
.. py:class:: ProcessingResult
This class is used to return results of processing of a payload by process_payload.
It includes a dictionary of the metadata (data), the database rowid, and the
state of the object as indicated by the ObjState enum.
.. py:attribute:: data
:type: dict[str, Any]
.. py:attribute:: obj_state
:type: ObjState
.. py:attribute:: rowid
:type: list[int]
.. py:data:: BETA_DB_VERSIONS
:value: [0, 1, 2, 3, 4, 5]
.. py:data:: CURRENT_DB_VERSION
:value: 15
.. py:data:: MIN_BATCH_SIZE
:value: 10
.. py:data:: MAX_BATCH_SIZE
:value: 1000
.. py:data:: POPULAR_TORRENTS_FRESHNESS_PERIOD
:value: 86400
.. py:data:: POPULAR_TORRENTS_COUNT
:value: 100
.. py:data:: sql_create_fts_table
:value: Multiline-String
.. raw:: html
Show Value
.. code-block:: python
"""
CREATE VIRTUAL TABLE IF NOT EXISTS FtsIndex USING FTS5
(title, content='ChannelNode', prefix = '2 3 4 5',
tokenize='porter unicode61 remove_diacritics 1');"""
.. raw:: html
.. py:data:: sql_add_fts_trigger_insert
:value: Multiline-String
.. raw:: html
Show Value
.. code-block:: python
"""
CREATE TRIGGER IF NOT EXISTS fts_ai AFTER INSERT ON ChannelNode
BEGIN
INSERT INTO FtsIndex(rowid, title) VALUES (new.rowid, new.title);
END;"""
.. raw:: html
.. py:data:: sql_add_fts_trigger_delete
:value: Multiline-String
.. raw:: html
Show Value
.. code-block:: python
"""
CREATE TRIGGER IF NOT EXISTS fts_ad AFTER DELETE ON ChannelNode
BEGIN
DELETE FROM FtsIndex WHERE rowid = old.rowid;
END;"""
.. raw:: html
.. py:data:: sql_add_fts_trigger_update
:value: Multiline-String
.. raw:: html
Show Value
.. code-block:: python
"""
CREATE TRIGGER IF NOT EXISTS fts_au AFTER UPDATE ON ChannelNode BEGIN
DELETE FROM FtsIndex WHERE rowid = old.rowid;
INSERT INTO FtsIndex(rowid, title) VALUES (new.rowid, new.title);
END;"""
.. raw:: html
.. py:data:: sql_add_torrentstate_trigger_after_insert
:value: Multiline-String
.. raw:: html
Show Value
.. code-block:: python
"""
CREATE TRIGGER IF NOT EXISTS torrentstate_ai AFTER INSERT ON TorrentState
BEGIN
UPDATE "TorrentState" SET has_data = (last_check > 0) WHERE rowid = new.rowid;
END;
"""
.. raw:: html
.. py:data:: sql_add_torrentstate_trigger_after_update
:value: Multiline-String
.. raw:: html
Show Value
.. code-block:: python
"""
CREATE TRIGGER IF NOT EXISTS torrentstate_au AFTER UPDATE ON TorrentState
BEGIN
UPDATE "TorrentState" SET has_data = (last_check > 0) WHERE rowid = new.rowid;
END;
"""
.. raw:: html
.. py:data:: sql_create_partial_index_torrentstate_last_check
:value: Multiline-String
.. raw:: html
Show Value
.. code-block:: python
"""
CREATE INDEX IF NOT EXISTS idx_torrentstate__last_check__partial
ON TorrentState (last_check, seeders, leechers, self_checked)
WHERE has_data = 1;
"""
.. raw:: html
.. py:class:: MetadataStore(db_filename: str, private_key: ipv8.keyvault.keys.PrivateKey, disable_sync: bool = False, notifier: tribler.core.notifier.Notifier | None = None, check_tables: bool = True, db_version: int = CURRENT_DB_VERSION)
Storage of metadata for channels and torrents.
.. py:attribute:: notifier
:value: None
.. py:attribute:: db_path
.. py:attribute:: my_key
.. py:attribute:: my_public_key_bin
.. py:attribute:: _logger
.. py:attribute:: _shutting_down
:value: False
.. py:attribute:: batch_size
:value: 10
.. py:attribute:: reference_timedelta
.. py:attribute:: sleep_on_external_thread
:value: 0.05
.. py:attribute:: db
.. py:attribute:: MiscData
.. py:attribute:: TrackerState
.. py:attribute:: TorrentState
.. py:attribute:: TorrentMetadata
.. py:method:: fast_integrity_check(remove_broken: bool = True) -> bool
Inspect the database file and return whether it was broken or not.
By default, this method also removes the given db file if it is broken. Set ``remove_broken`` to False if you
want to keep the broken file.
:returns: Whether the database file was broken (False if no errors occurred).
.. py:method:: set_value(key: str, value: str) -> None
Set a generic key to a value.
.. py:method:: get_value(key: str, default: str | None = None) -> str | None
Retrieve the value for a given key.
.. py:method:: drop_indexes() -> None
Drop the indices for this database.
.. py:method:: get_objects_to_create() -> list[pony.orm.core.Entity]
Get the objects that need to be created.
.. py:method:: get_db_file_size() -> int
Get the physical size on disk (always 0 for memory dbs).
.. py:method:: drop_fts_triggers() -> None
Drop the FTS triggers.
.. py:method:: create_fts_triggers() -> None
Create the FTS triggers.
.. py:method:: fill_fts_index() -> None
Insert the FTS indices.
.. py:method:: create_torrentstate_triggers() -> None
Create the torrent state triggers.
.. py:method:: shutdown() -> None
Disconnect the connection to the database.
.. py:method:: run_threaded(func: collections.abc.Callable, *args: Any, **kwargs) -> Any
:async:
Run ``func`` threaded and close DB connection at the end of the execution.
:param func: the function to be executed threaded
:param args: args for the function call
:param kwargs: kwargs for the function call
:return: a result of the func call.
.. py:method:: process_compressed_mdblob_threaded(compressed_data: bytes, **kwargs) -> list[ProcessingResult]
:async:
Decompress the given data in a thread and return a list of uncompressed results.
.. py:method:: process_compressed_mdblob(compressed_data: bytes, skip_personal_metadata_payload: bool = True) -> list[ProcessingResult]
Decompress the given data and return a list of uncompressed results.
.. py:method:: process_torrent_health(health: tribler.core.torrent_checker.healthdataclasses.HealthInfo) -> bool
Adds or updates information about a torrent health for the torrent with the specified infohash value.
:param health: a health info of a torrent
:return: True if a new TorrentState object was added
.. py:method:: process_squashed_mdblob(chunk_data: bytes, external_thread: bool = False, health_info: list[tuple[int, int, int]] | None = None, skip_personal_metadata_payload: bool = True) -> list[ProcessingResult]
Process raw concatenated payloads blob. This routine breaks the database access into smaller batches.
It uses a congestion-control like algorithm to determine the optimal batch size, targeting the
batch processing time value of self.reference_timedelta.
:param chunk_data: the blob itself, consists of one or more GigaChannel payloads concatenated together
:param external_thread: if this is set to True, we add some sleep between batches to allow other threads
to get the database lock. This is an ugly workaround for Python and asynchronous programming (locking)
imperfections. It only makes sense to use it when this routine runs on a non-reactor thread.
:param health_info: the health info to update a torrent with.
:param skip_personal_metadata_payload: don't process our own torrents.
:return: a list of tuples of (, )
.. py:method:: process_payload(payload: tribler.core.database.serialization.TorrentMetadataPayload, skip_personal_metadata_payload: bool = True) -> list[ProcessingResult]
Write a payload to our database (if necessary).
.. py:method:: get_num_torrents() -> int
Get the number of torrents in the database.
.. py:method:: search_keyword(query: str, origin_id: int | None = None) -> pony.orm.core.Query
Search for an FTS query, potentially restricted to a given origin id.
Requires FTS5 table "FtsIndex" to be generated and populated. FTS table is maintained automatically by SQL
triggers. BM25 ranking is embedded in FTS5.
.. py:method:: get_entries_query(metadata_type: int | None = None, channel_pk: bytes | None = None, hide_xxx: bool = False, origin_id: int | None = None, sort_by: str | None = None, sort_desc: bool = True, max_rowid: int | None = None, txt_filter: str | None = None, category: str | None = None, infohash: bytes | None = None, infohash_set: set[bytes] | None = None, id_: int | None = None, self_checked_torrent: bool | None = None, health_checked_after: int | None = None, popular: bool | None = None, tags: list[str] | None = None, **kwargs) -> pony.orm.core.Query
This method implements REST-friendly way to get entries from the database.
Warning! For Pony magic to work, iteration variable name (e.g. 'g') should be the same everywhere!
:return: PonyORM query object corresponding to the given params.
.. py:method:: get_entries_threaded(**kwargs) -> list[tribler.core.database.orm_bindings.torrent_metadata.TorrentMetadata]
:async:
Retrieve entries in a thread and return a list of results.
.. py:method:: get_entries(first: int = 1, last: int | None = None, **kwargs) -> list[tribler.core.database.orm_bindings.torrent_metadata.TorrentMetadata]
Get some torrents. Optionally sort the results by a specific field, or filter the channels based
on a keyword/whether you are subscribed to it.
:return: A list of class members
.. py:method:: get_total_count(**kwargs) -> int | None
Get total count of torrents that would be returned if there would be no pagination/limits/sort.
.. py:method:: get_entries_count(**kwargs) -> int | None
Get the count of torrents that would be returned if there would be no pagination/limits.
.. py:method:: get_max_rowid() -> int
Get the highest-known row id.
.. py:attribute:: fts_keyword_search_re
.. py:method:: get_auto_complete_terms(text: str, max_terms: int) -> list[str]
Get the auto-completion terms for a given query.