Compare commits

..

40 commits

Author SHA1 Message Date
d64c2398da
feat: add bot generated replies when it's being mentioned 2025-12-09 12:26:31 +01:00
3d44b44625
chore(release): upgrade metadata to v0.3.6 2025-06-06 14:55:56 +02:00
811fead29d
feat: add Quote.is_active field 2025-06-06 14:54:26 +02:00
7131930d8e
feat: add ANO_PREFIX bot response 2025-05-27 18:23:36 +02:00
4c2f85c105 chore(deps): update python docker tag to v3.13 2025-04-14 11:27:19 +00:00
13e21d53c1 chore(deps): lock file maintenance 2025-04-07 01:24:15 +00:00
983e4db02e
chore: add renovate lockFileMaintenance 2025-04-02 17:27:21 +02:00
dca90c8b22 Add renovate.json 2025-04-02 13:37:23 +00:00
4b7ffbc914
feat: implement remaining repo methods for chatbot and quote 2025-03-07 00:28:05 +01:00
95101a6573
fix: have uvicorn use Settings.port and Settings.host 2025-03-06 19:44:44 +01:00
c35741b8e1
chore: update charts to v0.3.2 2025-03-06 18:31:55 +01:00
740a1fc78f
fix: correctly execute make serve 2025-03-06 18:23:33 +01:00
e891d6fc1d
feat: add retry capabilities to the bot 2025-03-06 18:10:35 +01:00
839f67ad0a
feat: add UpdateVersionAction 2025-03-05 12:40:19 +01:00
152546982c
refactor: many changes
- Add missing actions and make a clear boundary between actions,
  services and nfra (i.e: actions shouldn't use stuff from infra/)
- Delete stuff not in use: tts, gtts, etc
- Add a ton of tests
2025-03-05 11:34:44 +01:00
b2185f4174
feat: remove !h and make the bot have an in-memory dict of greeted users instead of using the backoff service 2025-02-26 11:53:18 +01:00
48a3235323
fix: update token refresh to use object attributes instead of dict access 2025-02-24 14:31:46 +01:00
1452f2076b
fix: retrieve user profile after refreshing twitch creds 2025-02-24 11:29:12 +01:00
5616f0e76d
chore: update to v0.2.6 2025-02-13 10:55:31 +01:00
e428cd0672
fix: don't print the full author/user object when calling Quote().to_pretty() 2025-02-13 10:55:00 +01:00
378f314559
chore: update to v0.2.6 2025-02-13 10:23:18 +01:00
3058ca112d
feat: change QuoteStorerSvc to use the new quote repo instead of the legacy db object 2025-02-13 10:20:03 +01:00
75df191253
feat: add GetRandomQuoteAction 2025-02-13 09:52:15 +01:00
50900986fa
feat: revamp authentication -- remove twitch's tokens from our own wrapper token 2025-01-17 18:15:58 +01:00
3186afe96d
fix: fix logout flow which wasn't being triggered, remove useless html code 2024-12-19 18:50:05 +01:00
efac1cc33c
chore: update to v0.2.4 and remove useless code 2024-12-19 18:18:37 +01:00
3bc4e19de1
feat: add backoff service and some message reactions 2024-12-19 18:13:38 +01:00
2cad170eb3
chore: update to v0.2.3 2024-12-18 18:37:55 +01:00
9893d36be3
tests: add base tests 2024-12-18 18:27:46 +01:00
4c534de47b
feat: add migrations, api bot endpoints and revamp the whole twitch backend by making use of twitchio 2024-12-17 17:55:02 +01:00
8799bab900
feat: reduce execution queue length to 5 from 25 2024-12-12 15:40:17 +01:00
3aab3840af
feat: add channel name input validation 2024-12-12 14:51:32 +01:00
27f7bb0346
feat: add GET /tts/permalink 2024-12-12 13:10:25 +01:00
6b873348c7
feat: remove kivy frontend, add litestar 2024-12-12 12:22:34 +01:00
b71bedb62a
chore: update CHANGELOG 2024-11-05 16:44:09 +01:00
e01d9e4082
feat: add !q and !qadd commands 2024-11-05 16:42:15 +01:00
e152aa5f28
chore: update CHANGELOG 2024-11-01 16:11:33 +01:00
e574277839
feat: add config input for the automated generation option 2024-11-01 16:10:50 +01:00
3a33411dd9
feat: add Commands enum, delete useless commands, revamp the Settings class and the MarkovChain.message_handler method 2024-11-01 04:40:17 +01:00
29e1242591
chore: update CHANGELOG 2024-10-23 12:43:10 +02:00
348 changed files with 313123 additions and 1860 deletions

3
.envrc Normal file
View file

@ -0,0 +1,3 @@
source_url "https://raw.githubusercontent.com/cachix/devenv/82c0147677e510b247d8b9165c54f73d32dfd899/direnvrc" "sha256-7u4iDd1nZpxL4tCzmPG0dQgC5V+/44Ba+tHkPob1v2k="
use devenv

7
.gitignore vendored
View file

@ -88,7 +88,6 @@ celerybeat-schedule
celerybeat.pid
*.sage.py
.env
.venv
env/
venv/
ENV/
@ -110,3 +109,9 @@ reportlog.json
.ruff_cache/
.pdm.toml
requirements.txt
src/huesoporro/tts_files/
# Devenv
.devenv*
devenv.local.nix
# direnv
.direnv

56
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,56 @@
files: src|tests
exclude: ^$
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: trailing-whitespace
args: [ --markdown-linebreak-ext=md ]
- id: end-of-file-fixer
- id: check-ast
- id: check-added-large-files
- id: check-byte-order-marker
- id: check-case-conflict
- id: check-docstring-first
- id: check-merge-conflict
- id: check-toml
- id: debug-statements
- id: mixed-line-ending
args: [ --fix=lf ]
- repo: local
hooks:
- id: mypy
name: mypy
entry: uv run mypy --check-untyped-defs
language: system
types: [ python ]
exclude: LICENSE|helm
exclude_types:
- markdown
- css
- html
- id: ruff-format
name: uv run ruff format
language: system
entry: ruff format .
exclude: LICENSE|charts
exclude_types:
- markdown
- css
- html
- javascript
- id: ruff-check
name: ruff check
language: system
entry: uv run ruff check . --fix --exit-non-zero-on-fix
exclude: LICENSE|charts
exclude_types:
- markdown
- css
- html
- javascript

View file

@ -1 +1 @@
3.11
3.13

152
CHANGELOG
View file

@ -2,15 +2,165 @@
All notable changes to this project will be documented in this file.
## [unreleased]
## [0.3.6] - 2025-06-06
### 🚀 Features
- Add `Quote.is_active` field
## [0.3.5] - 2025-05-27
### 🚀 Features
- Implement remaining repo methods for chatbot and quote
- Add ANO_PREFIX bot response
### ⚙️ Miscellaneous Tasks
- Add renovate lockFileMaintenance
## [0.3.3] - 2025-03-06
### 🐛 Bug Fixes
- Have uvicorn use Settings.port and Settings.host
## [0.3.2] - 2025-03-06
### ⚙️ Miscellaneous Tasks
- Update charts to v0.3.2
## [0.3.1] - 2025-03-06
### 🐛 Bug Fixes
- Correctly execute `make serve`
## [0.3.0] - 2025-03-06
### 🚀 Features
- Remove !h and make the bot have an in-memory dict of greeted users instead of using the backoff service
- Add UpdateVersionAction
- Add retry capabilities to the bot
## [0.2.9] - 2025-02-24
### 🐛 Bug Fixes
- Update token refresh to use object attributes instead of dict access
## [0.2.8] - 2025-02-24
### 🐛 Bug Fixes
- Retrieve user profile after refreshing twitch creds
### ⚙️ Miscellaneous Tasks
- Update to v0.2.6
## [0.2.7] - 2025-02-13
### 🐛 Bug Fixes
- Don't print the full author/user object when calling Quote().to_pretty()
### ⚙️ Miscellaneous Tasks
- Update to v0.2.6
## [0.2.6] - 2025-02-13
### 🚀 Features
- Revamp authentication -- remove twitch's tokens from our own wrapper token
- Add GetRandomQuoteAction
- Change QuoteStorerSvc to use the new quote repo instead of the legacy db object
## [0.2.5] - 2024-12-19
### 🐛 Bug Fixes
- Fix logout flow which wasn't being triggered, remove useless html code
## [0.2.4] - 2024-12-19
### 🚀 Features
- Add backoff service and some message reactions
### ⚙️ Miscellaneous Tasks
- Update to v0.2.4 and remove useless code
## [0.2.3] - 2024-12-18
### 🧪 Testing
- Add base tests
### ⚙️ Miscellaneous Tasks
- Update to v0.2.3
## [0.2.2] - 2024-12-17
### 🚀 Features
- Add migrations, api bot endpoints and revamp the whole twitch backend by making use of twitchio
## [0.2.1] - 2024-12-12
### 🚀 Features
- Add GET /tts/permalink
- Add channel name input validation
- Reduce execution queue length to 5 from 25
## [0.2.0] - 2024-12-12
### 🚀 Features
- Remove kivy frontend, add litestar
### ⚙️ Miscellaneous Tasks
- Update CHANGELOG
## [0.1.2] - 2024-11-05
### 🚀 Features
- Add !q and !qadd commands
### ⚙️ Miscellaneous Tasks
- Update CHANGELOG
## [0.1.1] - 2024-11-01
### 🚀 Features
- Add Commands enum, delete useless commands, revamp the Settings class and the MarkovChain.message_handler method
- Add config input for the automated generation option
### ⚙️ Miscellaneous Tasks
- Update CHANGELOG
## [0.1.0] - 2024-10-23
### 🚀 Features
- Initial commit
- Add pyinstaller specfile
- Add onefile pyinstaller spec
### ⚙️ Miscellaneous Tasks
- Add authors pyproject metadata
- Add CHANGELOG
<!-- generated by git-cliff -->

46
Dockerfile Normal file
View file

@ -0,0 +1,46 @@
# hadolint ignore=DL3006,DL3007
FROM cgr.dev/chainguard/wolfi-base:latest AS base
SHELL ["/bin/ash", "-ex", "-c"]
ARG USERID=1000
ARG GROUPID=1000
ENV USERNAME="huesoporro"
ENV APP_HOME="/home/$USERNAME"
ENV APP_PATH="$APP_HOME"
ENV POETRY_VERSION=1.8.3
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONPATH="$APP_PATH"
ENV PATH="$APP_HOME/.local/bin:$PATH"
# hadolint ignore=DL3001,DL3008,DL3018
RUN apk add --no-cache make python3~=3.13 curl git \
&& adduser -S -u "$USERID" -h "$APP_HOME" "$USERNAME" \
&& mkdir -p "$APP_PATH" \
&& chown -R "$USERID:$GROUPID" "$APP_PATH"
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
USER "$USERNAME"
WORKDIR "$APP_PATH"
COPY --chown=$USERNAME pyproject.toml uv.lock Makefile README.md ./
COPY --chown=$USERNAME src/ src/
RUN uv sync
COPY --chown=$USERNAME migrations/ migrations/
FROM base AS serve
CMD ["make", "serve"]
FROM base AS migrate
CMD ["make", "migrate"]

661
LICENSE Normal file
View file

@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.

31
Makefile Normal file
View file

@ -0,0 +1,31 @@
PROJECT_NAME := "huesoporro"
PROJECT_TAG := "latest"
PROJECT_TARGET := "serve"
fmt:
uvx pre-commit run --all-files --color always
fmt--mypy:
uvx pre-commit run --all-files --color always mypy
fmt--add-noqa:
uvx ruff check --add-noqa .
fmt--autoupdate:
uvx pre-commit autoupdate
.PHONY: tests
tests:
uv run pytest --cov=src -vv tests
uv run coverage html
uv run coverage xml
serve:
uv run python src/apps/httpapi/litestar/main.py
build:
docker build . -t git.roboces.dev/catalin/$(PROJECT_NAME):$(PROJECT_TAG) --target $(PROJECT_TARGET)
migrate:
uv run caribou upgrade ~/.local/share/huesoporro/huesoporro.db migrations/

View file

@ -0,0 +1 @@
# huesoporro

View file

@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/

View file

@ -0,0 +1,6 @@
apiVersion: v2
appVersion: 0.3.7
description: A Helm chart for Kubernetes
name: huesoporro
type: application
version: 0.3.7

View file

@ -0,0 +1,22 @@
1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- range $host := .Values.ingress.hosts }}
{{- range .paths }}
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
{{- end }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "helm.fullname" . }})
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch its status by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "helm.fullname" . }}'
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "helm.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "helm.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}

View file

@ -0,0 +1,62 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "helm.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "helm.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "helm.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "helm.labels" -}}
helm.sh/chart: {{ include "helm.chart" . }}
{{ include "helm.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "helm.selectorLabels" -}}
app.kubernetes.io/name: {{ include "helm.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "helm.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "helm.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

View file

@ -0,0 +1,100 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "helm.fullname" . }}
labels:
{{- include "helm.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "helm.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "helm.labels" . | nindent 8 }}
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "helm.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- if and .Values.persistence.enabled .Values.persistence.volumeOwner.enabled }}
initContainers:
- name: volume-permissions
image: busybox
command: [ 'sh', '-c', 'chown -R {{ .Values.persistence.volumeOwner.uid }}:{{ .Values.persistence.volumeOwner.gid }} /data' ]
volumeMounts:
- name: data
mountPath: /data
securityContext:
runAsUser: 0
- name: migrate
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- make
- migrate
{{- if .Values.persistence.enabled }}
volumeMounts:
- name: data
mountPath: /home/huesoporro/.local/share/huesoporro
{{- end }}
{{- end }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.port }}
protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- if .Values.persistence.enabled }}
volumeMounts:
- name: data
mountPath: /home/huesoporro/.local/share/huesoporro
{{- end }}
securityContext:
runAsUser: {{ .Values.persistence.volumeOwner.uid }}
runAsGroup: {{ .Values.persistence.volumeOwner.gid }}
envFrom:
- secretRef:
name: {{ .Values.secret.existingSecretName }}
{{- if .Values.persistence.enabled }}
volumes:
- name: data
persistentVolumeClaim:
claimName: {{ include "helm.fullname" . }}-data
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}

View file

@ -0,0 +1,32 @@
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "helm.fullname" . }}
labels:
{{- include "helm.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "helm.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}

View file

@ -0,0 +1,43 @@
{{- if .Values.ingress.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "helm.fullname" . }}
labels:
{{- include "helm.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- with .Values.ingress.className }}
ingressClassName: {{ . }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
{{- with .pathType }}
pathType: {{ . }}
{{- end }}
backend:
service:
name: {{ include "helm.fullname" $ }}
port:
number: {{ $.Values.service.port }}
{{- end }}
{{- end }}
{{- end }}

View file

@ -0,0 +1,15 @@
{{- if .Values.persistence.enabled }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "helm.fullname" . }}-data
labels:
{{- include "helm.labels" . | nindent 4 }}
spec:
accessModes:
{{- toYaml .Values.persistence.accessModes | nindent 4 }}
resources:
requests:
storage: {{ .Values.persistence.size }}
storageClassName: {{ .Values.persistence.storageClassName }}
{{- end }}

View file

@ -0,0 +1,15 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "helm.fullname" . }}
labels:
{{- include "helm.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "helm.selectorLabels" . | nindent 4 }}

View file

@ -0,0 +1,13 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "helm.serviceAccountName" . }}
labels:
{{- include "helm.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
{{- end }}

View file

@ -0,0 +1,15 @@
apiVersion: v1
kind: Pod
metadata:
name: "{{ include "helm.fullname" . }}-test-connection"
labels:
{{- include "helm.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": test
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['{{ include "helm.fullname" . }}:{{ .Values.service.port }}']
restartPolicy: Never

View file

@ -0,0 +1,62 @@
affinity: {}
autoscaling:
enabled: false
maxReplicas: 100
minReplicas: 1
targetCPUUtilizationPercentage: 80
fullnameOverride: ''
image:
pullPolicy: Always
repository: git.roboces.dev/catalin/huesoporro
tag: 0.3.7
imagePullSecrets: []
ingress:
annotations: {}
className: ''
enabled: false
hosts:
- host: chart-example.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
livenessProbe:
httpGet:
path: /healthz
port: http
nameOverride: ''
nodeSelector: {}
persistence:
accessModes:
- ReadWriteMany
annotations: {}
enabled: false
size: 10Gi
storageClassName: default
volumeOwner:
enabled: true
gid: 1000
uid: 1000
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
readinessProbe:
httpGet:
path: /healthz
port: http
replicaCount: 1
resources: {}
secret:
existingSecretName: huesoporro-secrets
securityContext: {}
service:
port: 8000
type: LoadBalancer
serviceAccount:
annotations: {}
automount: true
create: true
name: ''
tolerations: []
volumeMounts: []
volumes: []

139
devenv.lock Normal file
View file

@ -0,0 +1,139 @@
{
"nodes": {
"devenv": {
"locked": {
"dir": "src/modules",
"lastModified": 1739362938,
"owner": "cachix",
"repo": "devenv",
"rev": "27276816caa1718f8b8e8d53d64cc18da059e101",
"type": "github"
},
"original": {
"dir": "src/modules",
"owner": "cachix",
"repo": "devenv",
"type": "github"
}
},
"flake-compat": {
"flake": false,
"locked": {
"lastModified": 1747046372,
"owner": "edolstra",
"repo": "flake-compat",
"rev": "9100a0f413b0c601e0533d1d94ffd501ce2e7885",
"type": "github"
},
"original": {
"owner": "edolstra",
"repo": "flake-compat",
"type": "github"
}
},
"flake-compat_2": {
"flake": false,
"locked": {
"lastModified": 1733328505,
"owner": "edolstra",
"repo": "flake-compat",
"rev": "ff81ac966bb2cae68946d5ed5fc4994f96d0ffec",
"type": "github"
},
"original": {
"owner": "edolstra",
"repo": "flake-compat",
"type": "github"
}
},
"git-hooks": {
"inputs": {
"flake-compat": "flake-compat",
"gitignore": "gitignore",
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1747372754,
"owner": "cachix",
"repo": "git-hooks.nix",
"rev": "80479b6ec16fefd9c1db3ea13aeb038c60530f46",
"type": "github"
},
"original": {
"owner": "cachix",
"repo": "git-hooks.nix",
"type": "github"
}
},
"gitignore": {
"inputs": {
"nixpkgs": [
"git-hooks",
"nixpkgs"
]
},
"locked": {
"lastModified": 1709087332,
"owner": "hercules-ci",
"repo": "gitignore.nix",
"rev": "637db329424fd7e46cf4185293b9cc8c88c95394",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "gitignore.nix",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1733477122,
"owner": "cachix",
"repo": "devenv-nixpkgs",
"rev": "7bd9e84d0452f6d2e63b6e6da29fe73fac951857",
"type": "github"
},
"original": {
"owner": "cachix",
"ref": "rolling",
"repo": "devenv-nixpkgs",
"type": "github"
}
},
"nixpkgs-python": {
"inputs": {
"flake-compat": "flake-compat_2",
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1733319315,
"owner": "cachix",
"repo": "nixpkgs-python",
"rev": "01263eeb28c09f143d59cd6b0b7c4cc8478efd48",
"type": "github"
},
"original": {
"owner": "cachix",
"repo": "nixpkgs-python",
"type": "github"
}
},
"root": {
"inputs": {
"devenv": "devenv",
"git-hooks": "git-hooks",
"nixpkgs": "nixpkgs",
"nixpkgs-python": "nixpkgs-python",
"pre-commit-hooks": [
"git-hooks"
]
}
}
},
"root": "root",
"version": 7
}

16
devenv.nix Normal file
View file

@ -0,0 +1,16 @@
{ pkgs, lib, config, inputs, ... }:
{
env.GREET = "devenv";
packages = [ pkgs.git ];
languages.python.enable = true;
languages.python.uv.enable = true;
languages.python.version = "3.12.8";
enterShell = ''
'';
dotenv.enable = true;
}

8
devenv.yaml Normal file
View file

@ -0,0 +1,8 @@
inputs:
nixpkgs:
url: github:cachix/devenv-nixpkgs/rolling
nixpkgs-python:
url: github:cachix/nixpkgs-python
inputs:
nixpkgs:
follows: nixpkgs

View file

@ -1,39 +0,0 @@
# -*- mode: python ; coding: utf-8 -*-
from kivy_deps import sdl2, glew
a = Analysis(
['src\\markovbot_gui\\main.py'],
pathex=[],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=0,
)
pyz = PYZ(a.pure)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
*[Tree(p) for p in (sdl2.dep_bins + glew.dep_bins)],
name='markovbot',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
upx_exclude=[],
runtime_tmpdir=None,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)

View file

@ -0,0 +1,29 @@
"""
This module contains a Caribou migration.
Migration Name: auth
Migration Version: 20241213175820
"""
def upgrade(connection):
# add your upgrade step here
sql = """
create table users
(
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
user varchar(255) NOT NULL UNIQUE,
access_token varchar(255) NOT NULL,
refresh_token varchar(255) NOT NULL,
expires_at TIMESTAMP NOT NULL,
last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,38 @@
"""
This module contains a Caribou migration.
Migration Name: quotes
Migration Version: 20241216204252
"""
def upgrade(connection):
# add your upgrade step here
sql = """
create table quotes
(
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
quote varchar(255) NOT NULL UNIQUE,
author varchar(255),
channel varchar(255),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
"""
connection.execute(sql)
sql = """
create table sentences
(
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
sentence varchar(255) NOT NULL UNIQUE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,28 @@
"""
This module contains a Caribou migration.
Migration Name: settings
Migration Version: 20241217000747
"""
def upgrade(connection):
sql = """
create table settings(
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
user_id VARCHAR(255) NOT NULL UNIQUE,
automatic_generation_timer INTENGER NOT NULL DEFAULT 300,
automatic_quote_timer INTEGER NOT NULL DEFAULT 500,
mods VARCHAR(255),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (user_id) REFERENCES users(user)
);
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,35 @@
"""
This module contains a Caribou migration.
Migration Name: sentences
Migration Version: 20241219191711
"""
def upgrade(connection):
# update table `sentences` to have a user_id row
# which references users.id
# and a channel VARCHAR(255) row
sql = """
DROP TABLE IF EXISTS sentences;
"""
connection.execute(sql)
connection.commit()
sql = """
CREATE TABLE sentences(
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
sentence VARCHAR(255) NOT NULL UNIQUE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
user_id VARCHAR(255) NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(id)
);
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,53 @@
"""
This module contains a Caribou migration.
Migration Name: user_external_auth
Migration Version: 20250112153541
"""
def upgrade(connection):
"""
- delete access_token, refresh_token, and expires_at from users
- add external_auth table which will store the external auths:
- type: twitch or discord
- credentials: JSON
"""
sql = """
ALTER TABLE users DROP COLUMN access_token;
"""
connection.execute(sql)
sql = """
ALTER TABLE users DROP COLUMN refresh_token;
"""
connection.execute(sql)
sql = """
ALTER TABLE users DROP COLUMN expires_at;
"""
connection.execute(sql)
sql = """
CREATE TABLE external_auth(
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
type VARCHAR(255) NOT NULL,
credentials JSON NOT NULL
);
"""
connection.execute(sql)
sql = """
CREATE TABLE user_external_auth(
user_id VARCHAR(255) NOT NULL,
external_auth_id INTEGER NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(id),
FOREIGN KEY (external_auth_id) REFERENCES external_auth(id)
);
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,35 @@
"""
This module contains a Caribou migration.
Migration Name: external_auth_json
Migration Version: 20250113142241
"""
def upgrade(connection):
"""remove tables:
- external_auth
- user_external_auth
add column to users table:
- external_auth JSON
"""
sql = """
DROP TABLE IF EXISTS external_auth;
"""
connection.execute(sql)
sql = """
DROP TABLE IF EXISTS user_external_auth;
"""
connection.execute(sql)
sql = """
ALTER TABLE users ADD COLUMN external_auth JSON;
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,19 @@
"""
This module contains a Caribou migration.
Migration Name: rename_settings_to_chatbot
Migration Version: 20250226120422
"""
def upgrade(connection):
sql = """
ALTER TABLE settings RENAME TO chatbot;
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,154 @@
"""
This module contains a Caribou migration.
Migration Name: uuids
Migration Version: 20250228112643
"""
import uuid
def upgrade(connection):
"""Several major upgrades
- Update all table's id columns to be UUIDs.
- Update the user's table `user` column to be named `username`
- Update the chatbot's table `user_id` column to reference the `id` column in the user's table,
thus changing the foreign key constraint.
"""
# First, create temporary tables with the new schema
connection.execute("""
CREATE TABLE users_new (
id TEXT not null PRIMARY KEY,
username varchar(255) not null UNIQUE,
last_updated_at TIMESTAMP default CURRENT_TIMESTAMP,
created_at TIMESTAMP default CURRENT_TIMESTAMP,
external_auth JSON
);
""")
# Fetch users to generate UUIDs in Python
users = connection.execute(
"SELECT user, last_updated_at, created_at, external_auth FROM users"
).fetchall()
# Copy data from old users table to new users table, using Python-generated UUIDs
for user in users:
new_uuid = uuid.uuid4().hex
# Safely handle JSON
connection.execute(
"""
INSERT INTO users_new (id, username, last_updated_at, created_at, external_auth)
VALUES (?, ?, ?, ?, ?)
""",
(new_uuid, user[0], user[1], user[2], user[3]),
)
# Create temporary quotes table with new schema
connection.execute("""
CREATE TABLE quotes_new (
id TEXT not null PRIMARY KEY,
quote varchar(255) not null UNIQUE,
author varchar(255),
channel varchar(255),
created_at TIMESTAMP default CURRENT_TIMESTAMP,
last_updated_at TIMESTAMP default CURRENT_TIMESTAMP
);
""")
# Fetch quotes to generate UUIDs in Python
quotes = connection.execute(
"SELECT quote, author, channel, created_at, last_updated_at FROM quotes"
).fetchall()
# Copy data from old quotes table to new quotes table, using Python-generated UUIDs
for quote in quotes:
new_uuid = uuid.uuid4().hex
connection.execute(
"""
INSERT INTO quotes_new (id, quote, author, channel, created_at, last_updated_at)
VALUES (?, ?, ?, ?, ?, ?)
""",
(new_uuid, quote[0], quote[1], quote[2], quote[3], quote[4]),
)
# Create mapping table to store the relationship between old user IDs and new UUIDs
connection.execute("""
CREATE TEMPORARY TABLE user_id_mapping (
old_user VARCHAR(255) not null,
new_id TEXT not null
);
""")
# Populate the mapping table
connection.execute("""
INSERT INTO user_id_mapping (old_user, new_id)
SELECT username, id FROM users_new;
""")
# Create temporary chatbot table with new schema
connection.execute("""
CREATE TABLE chatbot_new (
id TEXT not null PRIMARY KEY,
user_id TEXT not null UNIQUE,
automatic_generation_timer INTEGER default 300 not null,
automatic_quote_timer INTEGER default 500 not null,
mods VARCHAR(255),
created_at TIMESTAMP default CURRENT_TIMESTAMP,
last_updated_at TIMESTAMP default CURRENT_TIMESTAMP,
FOREIGN KEY (user_id) REFERENCES users_new (id)
);
""")
# Fetch chatbot data to generate UUIDs in Python
chatbots = connection.execute(
"SELECT user_id, automatic_generation_timer, automatic_quote_timer, mods, created_at, last_updated_at FROM chatbot"
).fetchall()
# Copy data from old chatbot table to new chatbot table with updated foreign key
for chatbot in chatbots:
# Get the new UUID for the user
user_mapping = connection.execute(
"SELECT new_id FROM user_id_mapping WHERE old_user = ?", (chatbot[0],)
).fetchone()
new_user_id = user_mapping[0] if user_mapping else None
if new_user_id:
new_chatbot_uuid = uuid.uuid4().hex
connection.execute(
"""
INSERT INTO chatbot_new (id, user_id, automatic_generation_timer, automatic_quote_timer, mods, created_at, last_updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?)
""",
(
new_chatbot_uuid,
new_user_id,
chatbot[1],
chatbot[2],
chatbot[3],
chatbot[4],
chatbot[5],
),
)
# Drop old tables
connection.execute("DROP TABLE chatbot;")
connection.execute("DROP TABLE quotes;")
connection.execute("DROP TABLE users;")
# Rename new tables to original names
connection.execute("ALTER TABLE users_new RENAME TO users;")
connection.execute("ALTER TABLE quotes_new RENAME TO quotes;")
connection.execute("ALTER TABLE chatbot_new RENAME TO chatbot;")
# Drop temporary mapping table
connection.execute("DROP TABLE user_id_mapping;")
# Commit all changes
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -0,0 +1,21 @@
"""
This module contains a Caribou migration.
Migration Name: active_quotes
Migration Version: 20250606143836
"""
def upgrade(connection):
# add `is_active` column to the `quotes` table
sql = """
ALTER TABLE quotes
ADD COLUMN is_active BOOLEAN DEFAULT TRUE;
"""
connection.execute(sql)
connection.commit()
def downgrade(connection):
# add your downgrade step here
pass

View file

@ -1,23 +1,80 @@
[project]
name = "markovbot-gui"
version = "0.1.0"
description = "Markov Chain Bot GUI"
name = "huesoporro"
version = "0.3.7"
description = "Misc Twitch bot"
readme = "README.md"
authors = [
{ name = "tomaarsen" },
{ name = "185504a9", email = "catalin@roboces.dev" }
]
requires-python = ">=3.11"
dependencies = [
"kivy[base]>=2.3.0",
"nltk>=3.9.1",
"pillow>=10.4.0",
"platformdirs>=4.3.6",
"pyinstaller>=6.11.0",
"twitchwebsocket>=1.2.1",
"pydantic>=2.9.2",
"pydantic-settings>=2.6.0",
"loguru>=0.7.2",
"gtts>=2.5.4",
"litestar[standard]>=2.13.0",
"httpx>=0.28.0",
"caribou>=0.4.1",
"aiosqlite>=0.20.0",
"pyjwt>=2.10.1",
"twitchio==2.10.0",
"redis>=5.2.1",
"pytz>=2024.2",
"discord-py>=2.4.0",
"tenacity>=9.0.0",
"uvicorn>=0.34.0",
"sniffio>=1.3.1",
]
[project.scripts]
huesoporro = "apps.cli.typer.main:app"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.uv]
dev-dependencies = [
"ruff>=0.7.0",
]
"mypy>=1.13.0",
"pytest>=8.3.4",
"pytest-asyncio>=0.25.0",
"ruff>=0.8.3",
"pytest-coverage>=0.0",
"polyfactory>=2.18.1",
"types-pyyaml>=6.0.12.20241230",
]
[[tool.mypy.overrides]]
module = [
"nltk",
"nltk.tokenize",
"nltk.tokenize.treebank",
"nltk.tokenize.destructive",
"TwitchWebsocket",
"tokenizer",
"caribou.migrate",
"twitchio",
"twitchio.ext",
"gtts",
"yt_dlp"
]
ignore_missing_imports = true
[tool.ruff.lint]
extend-select = [
"W", "C90", "I", "N", "UP", "S", "BLE", "B", "A", "COM", "C4", "DTZ", "T10", "EM", "ISC", "T20", "PT", "RSE", "RET",
"SIM", "PTH", "ERA", "PGH", "PL", "RUF", "FURB", "PERF"
]
extend-ignore = ["S101", "ISC002", "COM812", "ISC001", "EM101", "EM102"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"
[dependency-groups]
cli = [
"typer>=0.15.1",
"yt-dlp>=2025.1.26",
]

6
renovate.json5 Normal file
View file

@ -0,0 +1,6 @@
{
$schema: "https://docs.renovatebot.com/renovate-schema.json",
lockFileMaintenance: {
enabled: true,
},
}

View file

@ -0,0 +1,30 @@
from pathlib import Path
from loguru import logger
from typer import Typer
from huesoporro.actions.import_from_vod import ImportFromVODAction
from huesoporro.actions.misc.update_version_action import UpdateVersionAction
from huesoporro.settings import Settings
from huesoporro.svc.clean_cc_svc import CleanCCSvc
from huesoporro.svc.download_closed_captions import DownloadClosedCaptionsSvc
app = Typer()
@app.command()
def import_vod(channel_name: str, youtube_url: str, db_path: Path | None = None):
logger.info(f"Importing VOD closed captions for {channel_name} from {youtube_url}")
s = Settings.get(db_filepath=db_path)
import_from_vod_action = ImportFromVODAction(
download_closed_captions_svc=DownloadClosedCaptionsSvc(),
clean_cc_svc=CleanCCSvc(),
s=s,
)
for cc_filepath in import_from_vod_action.run(channel_name, youtube_url):
logger.info(f"Closed captions imported from {cc_filepath}")
@app.command()
def update_version(version: str, dry_run: bool = False):
UpdateVersionAction().run(version, dry_run)

View file

View file

View file

@ -0,0 +1,176 @@
from litestar import Request
from litestar.exceptions import HTTPException
from huesoporro.actions.chatbot.create_or_update_chatbot import (
CreateOrUpdateChatbotAction,
)
from huesoporro.actions.chatbot.get_chatbot_by_user_id import GetChatbotByUserIdAction
from huesoporro.actions.users.authenticate_user import AuthenticateUserAction
from huesoporro.actions.users.get_user_by_jwt import GetUserByJWTAction
from huesoporro.bot import BotsManager
from huesoporro.infra.authenticator import TwitchAuthenticator
from huesoporro.infra.repos import ChatbotRepo, UserRepo
from huesoporro.libs.db import MarkovDatabase
from huesoporro.models import Chatbot, User
from huesoporro.settings import Settings
from huesoporro.svc.chatbot_svcs import (
CreateChatbotSvc,
GetChatbotByUserIdSvc,
UpdateChatbotSvc,
)
from huesoporro.svc.store import SentenceStorerSvc
from huesoporro.svc.users_svcs import (
CreateUserSvc,
GetTwitchAuthByAuthCodeSvc,
GetUserByUsernameSvc,
IsValidTokenSvc,
RefreshTokenSvc,
UpdateUserSvc,
)
async def get_settings() -> Settings:
return Settings.get()
async def get_authenticator(s: Settings) -> TwitchAuthenticator:
return TwitchAuthenticator(s=s)
async def get_chatbot_repo(s: Settings):
return ChatbotRepo(s=s)
async def get_get_chatbot_by_user_id_svc(chatbot_repo: ChatbotRepo):
return GetChatbotByUserIdSvc(repo=chatbot_repo)
async def get_get_tokens_by_auth_code_svc(
twitch_authenticator: TwitchAuthenticator, s: Settings
):
return GetTwitchAuthByAuthCodeSvc(s=s, authenticator=twitch_authenticator)
async def get_create_chatbot_svc(chatbot_repo: ChatbotRepo):
return CreateChatbotSvc(repo=chatbot_repo)
async def get_user_repo(s: Settings):
return UserRepo(s=s)
async def get_create_user_svc(user_repo: UserRepo):
return CreateUserSvc(user_repo=user_repo)
async def get_update_user_svc(user_repo: UserRepo):
return UpdateUserSvc(user_repo=user_repo)
async def get_refresh_token_svc(twitch_authenticator: TwitchAuthenticator):
return RefreshTokenSvc(twitch_authenticator=twitch_authenticator)
async def get_is_valid_token_svc(twitch_authenticator: TwitchAuthenticator):
return IsValidTokenSvc(authenticator=twitch_authenticator)
async def get_get_user_by_username_svc(user_repo: UserRepo):
return GetUserByUsernameSvc(user_repo=user_repo)
async def get_get_user_by_jwt_action(
get_user_by_username_svc: GetUserByUsernameSvc,
update_user_svc: UpdateUserSvc,
is_valid_token_svc: IsValidTokenSvc,
refresh_token_svc: RefreshTokenSvc,
s: Settings,
):
return GetUserByJWTAction(
get_user_by_username_svc=get_user_by_username_svc,
update_user_svc=update_user_svc,
refresh_token_svc=refresh_token_svc,
is_valid_token_svc=is_valid_token_svc,
s=s,
)
async def authenticate(
request: Request, get_user_by_jwt_action: GetUserByJWTAction
) -> User:
token = request.query_params.get("huesoporro_token")
if token:
user = await get_user_by_jwt_action.run(token)
if not user:
raise HTTPException(detail="User does not exist", status_code=404)
return user
cookies = request.cookies.get("huesoporroAuth")
if cookies:
user = await get_user_by_jwt_action.run(cookies)
if not user:
raise HTTPException(detail="User does not exist", status_code=404)
return user
raise HTTPException(status_code=401, detail="Unauthorized")
async def get_sentences_storer_svc(db: MarkovDatabase):
return SentenceStorerSvc(db=db)
async def get_update_chatbot_svc(chatbot_repo: ChatbotRepo):
return UpdateChatbotSvc(repo=chatbot_repo)
async def get_create_or_update_chatbot_action(
create_chatbot_svc: CreateChatbotSvc,
update_chatbot_svc: UpdateChatbotSvc,
get_chatbot_by_user_id_svc: GetChatbotByUserIdSvc,
):
return CreateOrUpdateChatbotAction(
create_chatbot_svc=create_chatbot_svc,
update_chatbot_svc=update_chatbot_svc,
get_chatbot_by_user_id_svc=get_chatbot_by_user_id_svc,
)
async def get_get_chatbot_by_user_id_action(
get_chatbot_by_user_id_svc: GetChatbotByUserIdSvc,
):
return GetChatbotByUserIdAction(
get_chatbot_by_user_id_svc=get_chatbot_by_user_id_svc
)
async def get_authenticate_action(
s: Settings,
get_tokens_by_auth_code_svc: GetTwitchAuthByAuthCodeSvc,
get_user_by_username_svc: GetUserByUsernameSvc,
create_user_svc: CreateUserSvc,
update_user_svc: UpdateUserSvc,
):
return AuthenticateUserAction(
s=s,
get_tokens_by_auth_code_svc=get_tokens_by_auth_code_svc,
get_user_by_username_svc=get_user_by_username_svc,
create_user_svc=create_user_svc,
update_user_svc=update_user_svc,
)
async def get_bot_manager(s: Settings):
return BotsManager(s=s)
async def chatbot(
get_chatbot_by_user_id_action: GetChatbotByUserIdAction,
create_or_update_chatbot_action: CreateOrUpdateChatbotAction,
user: User,
) -> Chatbot:
cb = await get_chatbot_by_user_id_action.run(user_id=user.id)
if cb:
return cb
return await create_or_update_chatbot_action.run(
user_id=user.id,
)

View file

@ -0,0 +1,45 @@
import httpx
from litestar import MediaType, Request, Response
from litestar.exceptions import HTTPException
from litestar.response import Redirect
from litestar.status_codes import HTTP_500_INTERNAL_SERVER_ERROR
from loguru import logger
def http_exception_handler(_: Request, exc: HTTPException) -> Response:
status_code = getattr(exc, "status_code", HTTP_500_INTERNAL_SERVER_ERROR)
detail = getattr(exc, "detail", "")
if isinstance(exc, HTTPException) and (exc.status_code in [401, 403]):
logger.warning("User could not authenticate. Redirecting to /login page")
return Redirect("/login")
return Response(
media_type=MediaType.TEXT,
content=detail,
status_code=status_code,
)
def httpx_status_error_handler(_: Request, exc: httpx.HTTPStatusError):
logger.error(f"HTTPX error occurred: {exc}")
return Response(
media_type=MediaType.TEXT,
content=f"HTTPX error occurred: {exc}",
status_code=exc.response.status_code,
)
async def after_exception_handler(exc: Exception, scope: "Scope") -> None: # type: ignore[name-defined] # noqa: F821
"""Hook function that will be invoked after each exception."""
state = scope["app"].state
if not hasattr(state, "error_count"):
state.error_count = 1
else:
state.error_count += 1
logger.error(
f"an exception of type {type(exc).__name__} has occurred for requested path {scope['path']} and the application error count is {state.error_count}.",
)
import traceback
traceback.print_exc()

View file

@ -0,0 +1,122 @@
import httpx
import uvicorn
from litestar import Litestar, get
from litestar.contrib.jinja import JinjaTemplateEngine
from litestar.di import Provide
from litestar.exceptions import HTTPException
from litestar.static_files import StaticFilesConfig
from litestar.template import TemplateConfig
from apps.httpapi.litestar.dependencies import (
authenticate,
get_authenticate_action,
get_authenticator,
get_bot_manager,
get_chatbot_repo,
get_create_chatbot_svc,
get_create_or_update_chatbot_action,
get_create_user_svc,
get_get_chatbot_by_user_id_action,
get_get_chatbot_by_user_id_svc,
get_get_tokens_by_auth_code_svc,
get_get_user_by_jwt_action,
get_get_user_by_username_svc,
get_is_valid_token_svc,
get_refresh_token_svc,
get_sentences_storer_svc,
get_settings,
get_update_chatbot_svc,
get_update_user_svc,
get_user_repo,
)
from apps.httpapi.litestar.errors import (
after_exception_handler,
http_exception_handler,
httpx_status_error_handler,
)
from apps.httpapi.litestar.routes.api import (
get_bot_settings,
get_bot_status,
get_index,
get_tts_overlay,
get_tts_permalink,
manage_bot,
save_bot_settings,
)
from apps.httpapi.litestar.routes.auth import get_code, login
from huesoporro.settings import Settings
@get("/healthz")
async def get_health() -> dict:
return {"status": "ok"}
def create_app():
return Litestar(
route_handlers=[
get_health,
login,
get_index,
get_tts_overlay,
get_tts_permalink,
get_code,
manage_bot,
get_bot_status,
save_bot_settings,
get_bot_settings,
],
static_files_config=(
StaticFilesConfig(
path="/tts_files",
directories=[Settings.get().tts_cache_path],
),
StaticFilesConfig(
path="static",
directories=[Settings.get().static_files_path],
),
),
template_config=TemplateConfig(
directory=Settings.get().templates_files_path,
engine=JinjaTemplateEngine,
),
exception_handlers={
HTTPException: http_exception_handler,
httpx.HTTPStatusError: httpx_status_error_handler,
},
after_exception=[after_exception_handler],
dependencies={
"s": Provide(get_settings, use_cache=True),
"a": Provide(get_authenticator, use_cache=True),
"user": Provide(authenticate),
"bm": Provide(get_bot_manager, use_cache=True),
"sss": Provide(get_sentences_storer_svc),
"twitch_authenticator": Provide(get_authenticator),
"authenticate_action": Provide(get_authenticate_action),
"user_repo": Provide(get_user_repo),
"chatbot_repo": Provide(get_chatbot_repo),
"create_user_svc": Provide(get_create_user_svc),
"update_chatbot_svc": Provide(get_update_chatbot_svc),
"update_user_svc": Provide(get_update_user_svc),
"create_chatbot_svc": Provide(get_create_chatbot_svc),
"refresh_token_svc": Provide(get_refresh_token_svc),
"is_valid_token_svc": Provide(get_is_valid_token_svc),
"get_user_by_username_svc": Provide(get_get_user_by_username_svc),
"get_chatbot_by_user_id_svc": Provide(get_get_chatbot_by_user_id_svc),
"get_tokens_by_auth_code_svc": Provide(get_get_tokens_by_auth_code_svc),
"get_user_by_jwt_action": Provide(get_get_user_by_jwt_action),
"get_chatbot_by_user_id_action": Provide(get_get_chatbot_by_user_id_action),
"create_or_update_chatbot_action": Provide(
get_create_or_update_chatbot_action
),
},
)
app = create_app()
if __name__ == "__main__":
s = Settings.get()
config = uvicorn.Config("main:app", host=s.host, port=s.port, log_level="info")
server = uvicorn.Server(config)
server.run()

View file

@ -0,0 +1,120 @@
from typing import Literal
from litestar import MediaType, Response, get, put
from litestar.datastructures import UploadFile
from litestar.response import Template
from pydantic import BaseModel, ConfigDict
from huesoporro.actions.chatbot.create_or_update_chatbot import (
CreateOrUpdateChatbotAction,
)
from huesoporro.actions.chatbot.get_chatbot_by_user_id import GetChatbotByUserIdAction
from huesoporro.bot import BotsManager
from huesoporro.models import Chatbot, User
class ManageBotDTO(BaseModel):
command: Literal["start", "stop"]
channel_name: str | None = None
class ImportTextFileDTO(BaseModel):
file: UploadFile
channel_name: str
model_config = ConfigDict(arbitrary_types_allowed=True)
class UpdateChatbotDTO(BaseModel):
automatic_generation_timer: int = 300
automatic_quote_timer: int = 500
mods: list[str]
@get(
"/tts",
media_type=MediaType.HTML,
)
async def get_tts_overlay(user: User) -> Template:
return Template(template_name="tts.html")
@get(
"/tts/permalink",
media_type=MediaType.HTML,
)
async def get_tts_permalink(access_token: str) -> Template:
"""Handler for the /tts permalink endpoint to be used by apps that can only give the authentication as a query
param and not as a cookie, i.e. OBS"""
return Template(
template_name="tts.html",
)
@get(
"/",
media_type=MediaType.HTML,
)
async def get_index(
user: User, get_chatbot_by_user_id_action: GetChatbotByUserIdAction
) -> Template:
chatbot_settings = await get_chatbot_by_user_id_action.run(user_id=user.id)
return Template(
template_name="index.html",
context=chatbot_settings.model_dump() if chatbot_settings else {},
)
@put("/api/v1/bot")
async def manage_bot(
user: User,
data: ManageBotDTO,
create_or_update_chatbot_action: CreateOrUpdateChatbotAction,
get_chatbot_by_user_id_action: GetChatbotByUserIdAction,
bm: BotsManager,
) -> Response:
chatbot = await get_chatbot_by_user_id_action.run(
user_id=user.id
) or await create_or_update_chatbot_action.run(
user_id=user.id,
)
if data.command == "start":
if not data.channel_name:
return Response({"message": "Channel name is required"}, status_code=400)
bm.add_bot(user, data.channel_name, chatbot=chatbot) # type: ignore[arg-type]
if user.username in bm.bots:
await bm.run_user_bot(user)
return Response({"message": "Bot started"})
if data.command == "stop" and user.username in bm.bots:
await bm.stop_user_bot(user)
return Response({"message": "Bot stopped"})
return Response({"message": "Invalid command"}, status_code=400)
@get("/api/v1/bot")
async def get_bot_status(user: User, bm: BotsManager) -> dict:
if user.username not in bm.bots:
return {"status": "ko"}
return {"status": "ok"}
@get("/api/v1/bot/settings")
async def get_bot_settings(chatbot: Chatbot) -> Chatbot:
return chatbot
@put("/api/v1/bot/settings")
async def save_bot_settings(
user: User,
data: UpdateChatbotDTO,
create_or_update_chatbot_action: CreateOrUpdateChatbotAction,
) -> dict:
await create_or_update_chatbot_action.run(
user_id=user.id,
automatic_generation_timer=data.automatic_generation_timer,
automatic_quote_timer=data.automatic_quote_timer,
mods=data.mods,
)
return {"status": "ok"}

View file

@ -0,0 +1,42 @@
import secrets
from litestar import MediaType, get
from litestar.datastructures.cookie import Cookie
from litestar.response import Redirect, Template
from huesoporro.actions.users.authenticate_user import AuthenticateUserAction
from huesoporro.settings import Settings
@get(path="/o/code")
async def get_code(code: str, authenticate_action: AuthenticateUserAction) -> Redirect:
user = await authenticate_action.run(code)
token = user.encode()
return Redirect(
"/",
cookies=[
Cookie(
key="huesoporroAuth",
value=token,
expires=604800, # 1 week
)
],
)
@get(
"/login",
media_type=MediaType.HTML,
)
async def login(s: Settings) -> Template:
scopes = "+".join(s.twitch_scopes)
return Template(
"login.html",
context={
"twitch_login_url": "https://id.twitch.tv/oauth2/authorize?response_type=code"
f"&client_id={s.twitch_client_id}"
f"&redirect_uri={s.server_hostname}o/code"
f"&scope={scopes}"
f"&state={secrets.token_urlsafe(32)}"
},
)

View file

View file

View file

@ -0,0 +1,40 @@
import uuid
from uuid import UUID
from pydantic import BaseModel
from huesoporro.models import Chatbot
from huesoporro.svc.chatbot_svcs import (
CreateChatbotSvc,
GetChatbotByUserIdSvc,
UpdateChatbotSvc,
)
class CreateOrUpdateChatbotAction(BaseModel):
create_chatbot_svc: CreateChatbotSvc
update_chatbot_svc: UpdateChatbotSvc
get_chatbot_by_user_id_svc: GetChatbotByUserIdSvc
async def run(
self,
user_id: UUID,
automatic_generation_timer: int = 300,
automatic_quote_timer: int = 500,
mods: list[str] | None = None,
) -> Chatbot:
mods = mods or []
chatbot = await self.get_chatbot_by_user_id_svc.run(user_id=user_id)
if chatbot:
chatbot.automatic_generation_timer = automatic_generation_timer
chatbot.automatic_quote_timer = automatic_quote_timer
chatbot.mods = mods
return await self.update_chatbot_svc.run(chatbot=chatbot)
chatbot = Chatbot(
id=uuid.uuid4(),
user_id=user_id,
automatic_generation_timer=automatic_generation_timer,
automatic_quote_timer=automatic_quote_timer,
mods=mods,
)
return await self.create_chatbot_svc.run(chatbot=chatbot)

View file

@ -0,0 +1,16 @@
from uuid import UUID
from pydantic import BaseModel
from huesoporro.models import Chatbot
from huesoporro.svc.chatbot_svcs import GetChatbotByUserIdSvc
class GetChatbotByUserIdAction(BaseModel):
get_chatbot_by_user_id_svc: GetChatbotByUserIdSvc
async def run(
self,
user_id: UUID,
) -> Chatbot | None:
return await self.get_chatbot_by_user_id_svc.run(user_id=user_id)

View file

@ -0,0 +1,34 @@
from collections.abc import Generator
from pathlib import Path
from pydantic import BaseModel, Field
from huesoporro.libs.db import MarkovDatabase
from huesoporro.settings import Settings
from huesoporro.svc.clean_cc_svc import CleanCCSvc
from huesoporro.svc.download_closed_captions import DownloadClosedCaptionsSvc
from huesoporro.svc.store import SentenceStorerSvc
class ImportFromVODAction(BaseModel):
download_closed_captions_svc: DownloadClosedCaptionsSvc
clean_cc_svc: CleanCCSvc
s: Settings = Field(default_factory=Settings.get)
ignore_lines: set[str] = {
"WEBVTT",
"Kind: captions",
"Language: en",
"Language: es",
}
def run(self, channel_name: str, youtube_url: str) -> Generator[Path, None, None]:
for cc_filepath in self.download_closed_captions_svc.run(youtube_url):
storer_svc = SentenceStorerSvc(
db=MarkovDatabase(channel=channel_name, settings=self.s)
)
for line in self.clean_cc_svc.run(cc_filepath):
if line and line not in self.ignore_lines:
storer_svc.store_sentence(line.strip())
yield cc_filepath

View file

View file

@ -0,0 +1,198 @@
import re
from collections.abc import Callable
from difflib import unified_diff
from pathlib import Path
import yaml
from loguru import logger
from pydantic import BaseModel, ConfigDict
from rich import print # noqa: A004
from rich.console import Console
from rich.panel import Panel
from rich.syntax import Syntax
class UpdateVersionAction(BaseModel):
project_root: Path = Path(__file__).parents[4]
files_to_update: dict[str, Callable]
console: Console = Console()
model_config = ConfigDict(arbitrary_types_allowed=True)
def __init__(self, **data):
files_to_update = {
"pyproject.toml": self._update_pyproject_toml,
"charts/huesoporro/values.yaml": self._update_values_yaml,
"charts/huesoporro/Chart.yaml": self._update_chart_yaml,
}
super().__init__(**data, files_to_update=files_to_update)
def _read_file(self, filepath: Path) -> str:
"""
Read the contents of a file.
Args:
filepath (Path): Path to the file to read.
Returns:
str: File contents
"""
with filepath.open("r") as f:
return f.read()
def _write_file(self, filepath: Path, content: str):
"""
Write content to a file.
Args:
filepath (Path): Path to the file to write.
content (str): Content to write to the file.
"""
with filepath.open("w") as f:
f.write(content)
def _update_pyproject_toml(self, filepath: Path, new_version: str) -> str:
"""
Update version in pyproject.toml.
Args:
filepath (Path): Path to pyproject.toml
new_version (str): New version to set
Returns:
str: Updated file content
"""
content = self._read_file(filepath)
version_pattern = r'(version\s*=\s*)[\'"](.+?)[\'"]'
return re.sub(version_pattern, rf'\1"{new_version}"', content)
def _update_values_yaml(self, filepath: Path, new_version: str) -> str:
"""
Update image tag in values.yaml.
Args:
filepath (Path): Path to values.yaml
new_version (str): New version to set
Returns:
str: Updated file content
"""
with filepath.open("r") as file:
values = yaml.safe_load(file)
# Assumes image.tag exists in the values.yaml
values["image"]["tag"] = new_version
return yaml.dump(values, default_flow_style=False)
def _update_chart_yaml(self, filepath: Path, new_version: str) -> str:
"""
Update version and appVersion in Chart.yaml.
Args:
filepath (Path): Path to Chart.yaml
new_version (str): New version to set
Returns:
str: Updated file content
"""
with filepath.open("r") as file:
chart_data = yaml.safe_load(file)
chart_data["version"] = new_version
chart_data["appVersion"] = new_version
return yaml.dump(chart_data, default_flow_style=False)
def _generate_diff(self, original: str, updated: str, filename: str) -> str:
"""
Generate a unified diff between original and updated content.
Args:
original (str): Original file content
updated (str): Updated file content
filename (str): Name of the file
Returns:
str: Unified diff representation
"""
# Split content into lines
original_lines = original.splitlines(keepends=True)
updated_lines = updated.splitlines(keepends=True)
# Generate unified diff
diff_lines = list(
unified_diff(
original_lines,
updated_lines,
fromfile=f"a/{filename}",
tofile=f"b/{filename}",
lineterm="",
)
)
return "\n".join(diff_lines)
def _rich_display_diff(self, diff: str):
"""
Display diff using rich for colorful output.
Args:
diff (str): Unified diff to display
"""
if not diff:
return
# Use Syntax for syntax highlighting
syntax = Syntax(diff, "diff", theme="ansi_dark")
# Create a panel with the diff
panel = Panel(
syntax, title="Version Update Diff", border_style="cyan", expand=False
)
# Display the panel
self.console.print(panel)
def run(self, new_version: str, dry_run: bool = False):
"""
Update version across all specified files.
Args:
new_version (str): New version to set
dry_run (bool): Dry run mode with diff display
"""
for relative_path, update_func in self.files_to_update.items():
filepath = self.project_root / relative_path
if not filepath.exists():
logger.warning(f"Warning: {filepath} not found. Skipping.")
continue
try:
# Read original content
original_content = self._read_file(filepath)
# Generate updated content
updated_content = update_func(filepath, new_version)
if dry_run:
# Generate and display diff
diff = self._generate_diff(
original_content, updated_content, str(relative_path)
)
# Display the diff
if diff:
print(f"\nDiff for {relative_path}:")
self._rich_display_diff(diff)
else:
# Write updated content
self._write_file(filepath, updated_content)
print(f"Updated {relative_path}")
except Exception as exc: # noqa: BLE001
logger.error(f"Error updating {relative_path}: {exc}")
if dry_run:
print("\nDry run complete. No files were modified.")

View file

@ -0,0 +1,35 @@
import datetime
import uuid
from pydantic import BaseModel
from huesoporro.models import Quote, User
from huesoporro.svc.is_mod import IsModSvc
from huesoporro.svc.quotes_svcs import CreateQuoteSvc
class CreateQuoteAction(BaseModel):
create_quote_svc: CreateQuoteSvc
is_mod_svc: IsModSvc
async def run( # noqa: PLR0913
self,
user: User,
channel: str,
quote: str,
author: str,
username: str,
is_active: bool = True,
) -> Quote | None:
if not await self.is_mod_svc.run(user=user, username=username, channel=channel):
return None
new_quote = Quote(
id=uuid.uuid4(),
quote=quote,
author=author,
channel_name=channel,
created_at=datetime.datetime.now(datetime.UTC),
is_active=is_active,
last_updated_at=datetime.datetime.now(datetime.UTC),
)
return await self.create_quote_svc.run(new_quote)

View file

@ -0,0 +1,11 @@
from pydantic import BaseModel
from huesoporro.models import Quote
from huesoporro.svc.quotes_svcs import GetRandomQuoteSvc
class GetRandomQuoteAction(BaseModel):
get_random_quote_svc: GetRandomQuoteSvc
async def run(self, channel_name: str) -> Quote | None:
return await self.get_random_quote_svc.run(channel_name=channel_name)

View file

View file

@ -0,0 +1,38 @@
import uuid
from pydantic import BaseModel
from huesoporro.models import User
from huesoporro.settings import Settings
from huesoporro.svc.users_svcs import (
CreateUserSvc,
GetTwitchAuthByAuthCodeSvc,
GetUserByUsernameSvc,
UpdateUserSvc,
)
class AuthenticateUserAction(BaseModel):
get_tokens_by_auth_code_svc: GetTwitchAuthByAuthCodeSvc
get_user_by_username_svc: GetUserByUsernameSvc
create_user_svc: CreateUserSvc
update_user_svc: UpdateUserSvc
s: Settings
async def run(
self,
auth_code: str,
) -> User:
auth = await self.get_tokens_by_auth_code_svc.run(auth_code=auth_code)
username = auth.userinfo["preferred_username"]
user = await self.get_user_by_username_svc.run(username=username)
if user:
user.external_auth = {"twitch": auth}
await self.update_user_svc.run(user)
return user
user = User(
id=uuid.uuid4(),
username=username,
external_auth={"twitch": auth},
)
return await self.create_user_svc.run(user=user)

View file

@ -0,0 +1,40 @@
from loguru import logger
from pydantic import BaseModel
from huesoporro.models import User
from huesoporro.settings import Settings
from huesoporro.svc.users_svcs import (
GetUserByUsernameSvc,
IsValidTokenSvc,
RefreshTokenSvc,
UpdateUserSvc,
)
class GetUserByJWTAction(BaseModel):
get_user_by_username_svc: GetUserByUsernameSvc
update_user_svc: UpdateUserSvc
refresh_token_svc: RefreshTokenSvc
is_valid_token_svc: IsValidTokenSvc
s: Settings
async def run(
self,
jwt_token: str,
) -> User | None:
user_data = User.decode(jwt_token, settings=self.s)
username = user_data["username"]
user = await self.get_user_by_username_svc.run(username=username)
if not user:
raise ValueError(f"User {username} not found")
if await self.is_valid_token_svc.run(user=user):
logger.info(
f"User {username} has a correct twitch authentication token, returning user"
)
return user
logger.info(
f"User {username} has an invalid twitch authentication token, refreshing it"
)
user = await self.refresh_token_svc.run(user=user)
return await self.update_user_svc.run(user=user)

View file

@ -0,0 +1,26 @@
from pydantic import BaseModel
from huesoporro.models import User
from huesoporro.settings import Settings
from huesoporro.svc.users_svcs import (
GetUserByUsernameSvc,
IsValidTokenSvc,
RefreshTokenSvc,
UpdateUserSvc,
)
class RefreshUserJwtAction(BaseModel):
get_user_by_username_svc: GetUserByUsernameSvc
update_user_svc: UpdateUserSvc
refresh_token_svc: RefreshTokenSvc
is_valid_token_svc: IsValidTokenSvc
s: Settings
async def run(self, user: User) -> User | None:
"""Return None if the user has a valid token, otherwise refresh it and return the new token"""
if await self.is_valid_token_svc.run(user=user):
return None
user = await self.refresh_token_svc.run(user=user)
return await self.update_user_svc.run(user=user)

380
src/huesoporro/bot.py Normal file
View file

@ -0,0 +1,380 @@
import asyncio
import random
from collections.abc import Callable
from enum import StrEnum
from typing import ClassVar
from loguru import logger
from tenacity import (
retry,
retry_if_exception_type,
stop_after_attempt,
wait_exponential,
)
from twitchio import Channel, Message
from twitchio.ext import commands, routines
from huesoporro.actions.quotes.create_quote_action import CreateQuoteAction
from huesoporro.actions.quotes.get_random_quote import GetRandomQuoteAction
from huesoporro.infra.repos import ChatbotRepo, QuoteRepo
from huesoporro.libs.db import MarkovDatabase
from huesoporro.models import Chatbot, User
from huesoporro.settings import Settings
from huesoporro.svc.backoff_service import BackoffService
from huesoporro.svc.generate import SentenceGeneratorSvc
from huesoporro.svc.hello import get_hello_generator_svc
from huesoporro.svc.is_mod import IsModSvc
from huesoporro.svc.quotes_svcs import CreateQuoteSvc, GetRandomQuoteSvc
from huesoporro.svc.store import SentenceStorerSvc
class Bot(commands.Bot):
def __init__dependencies(self, channel: str, settings: Settings):
self.quote_repo = QuoteRepo(s=settings)
self.chatbot_repo = ChatbotRepo(s=settings)
self.get_random_quote_action = GetRandomQuoteAction(
get_random_quote_svc=GetRandomQuoteSvc(repo=self.quote_repo)
)
self.create_quote_action = CreateQuoteAction(
create_quote_svc=CreateQuoteSvc(repo=self.quote_repo),
is_mod_svc=IsModSvc(repo=self.chatbot_repo),
)
self.generate_svc = SentenceGeneratorSvc(db=MarkovDatabase(channel=channel))
self.hello_svc = get_hello_generator_svc()
def __init__(self, user: User, chatbot: Chatbot, channel: str, settings: Settings):
super().__init__(
token=user.twitch_access_token, prefix="!", initial_channels=[channel]
)
self.__init__dependencies(channel=channel, settings=settings)
self.channel = channel
self.user = user
self.chatbot = chatbot
self.quote_routine = routines.routine(
seconds=chatbot.automatic_quote_timer, wait_first=True
)(self.send_quote)
self.generation_routine = routines.routine(
seconds=chatbot.automatic_generation_timer, wait_first=True
)(self.send_generation)
async def event_ready(self):
logger.info(f"Logged in as {self.nick}")
logger.info(f"User id is {self.user_id}")
@commands.command(aliases=["g"])
async def generate(self, ctx: commands.Context, *, words: str | None = None):
sentence = await self.generate_svc.run(words)
if not sentence:
logger.warning(
f"Could not generate sentence for {words or 'no words provided'}"
)
return
await ctx.send(sentence)
# Wait for the specified time
await asyncio.sleep(60)
@commands.command(aliases=["qadd"])
async def add_quote(self, ctx: commands.Context, *, quote: str):
# extract author from quote; the author is the last word
quote, author = quote.rsplit(" ", 1)
new_quote = await self.create_quote_action.run(
user=self.user,
channel=self.channel,
quote=quote,
author=author,
username=ctx.author.name,
)
if new_quote:
await ctx.send(new_quote.as_pretty_saved())
else:
await ctx.send(f"@{ctx.author.name} no tienes permisos para añadir citas")
@commands.command(aliases=["q", "quote"])
async def get_random_quote(self, ctx: commands.Context):
quote = await self.get_random_quote_action.run(channel_name=self.channel)
if quote:
await ctx.send(quote.as_pretty())
def get_channel_conn(self) -> Channel:
return Channel(name=self.channel, websocket=self._connection)
async def send_quote(self):
quote = await self.get_random_quote_action.run(channel_name=self.channel)
if quote:
channel = self.get_channel_conn()
if channel:
logger.info(f"Sending random quote {quote.quote}")
await channel.send(quote.quote)
async def send_generation(self):
sentence = await self.generate_svc.run()
if not sentence:
return
channel = self.get_channel_conn()
logger.info(f"Sending generated sentence {sentence}")
await channel.send(sentence)
def start_routines(self):
if self.chatbot.automatic_quote_timer > 0:
logger.info("Starting quote routine")
self.quote_routine.start(stop_on_error=False)
if self.chatbot.automatic_generation_timer > 0:
logger.info("Starting generation routine")
self.generation_routine.start(stop_on_error=False)
def stop_routines(self):
logger.info("Stopping routines")
self.quote_routine.cancel()
self.generation_routine.cancel()
class HelloMessagesCog(commands.Cog):
hello_patterns: ClassVar[list[str]] = ["hola", "HOLA", "hiii", "ayo"]
def __init__(self, bot):
self.bot = bot
self.hello_svc = get_hello_generator_svc()
@commands.Cog.event()
async def event_message(self, message):
if not message.author:
return
if message.content in self.hello_patterns:
hello = self.hello_svc.run(message.author.name)
if hello:
await message.channel.send(hello)
class MessageType(StrEnum):
COMMAND = "COMMAND"
HELLO = "HELLO"
YES = "YES"
WHAT = "WHAT"
LAUGH = "LAUGH"
ANO_SUFFIX = "ANO_SUFFIX"
OTHER = "OTHER"
class MessageHandler:
"""Handles different types of messages with their corresponding responses"""
def __init__(self, channel_send_func: Callable):
self.laugh_patterns = [
"om",
"KEK",
"LuL",
"LUL",
"OMEGALUL",
"kek",
"keking",
"KEKW",
"OMEGADANCEBUTFAST",
"xdd",
"xdding",
]
self.ano_suffix_reply_patterns = [
"me la agarras con la mano. venga, tira",
"me la agarras con la mano, espabila",
"me la agarras con la mano y te falta calle",
"vegetasmile",
]
self.send = channel_send_func
def get_message_type(self, content: str) -> MessageType:
"""Determines the type of message based on its content"""
if content.startswith("!"):
return MessageType.COMMAND
if content in ["Yes", "yes"]:
return MessageType.YES
if content.startswith("WHAT"):
return MessageType.WHAT
if content.endswith("ano") and len(content) > 3: # noqa: PLR2004
return MessageType.ANO_SUFFIX
if content in self.laugh_patterns:
return MessageType.LAUGH
return MessageType.OTHER
def handle_laugh(self) -> str:
return random.choice(self.laugh_patterns) # noqa: S311
def handle_ano_suffix(self) -> str:
return random.choice(self.ano_suffix_reply_patterns) # noqa: S311
class SaveMessagesCog(commands.Cog):
def __init__(self, bot: Bot):
self.bot = bot
self.store_svc = SentenceStorerSvc(db=MarkovDatabase(channel=bot.channel))
self.generate_svc = SentenceGeneratorSvc(db=MarkovDatabase(channel=bot.channel))
self.backoff_svc = BackoffService()
self.message_handler = MessageHandler(self._send_message)
self.send_functions = {
MessageType.YES: self._create_typed_send("yes"),
MessageType.WHAT: self._create_typed_send("what"),
MessageType.LAUGH: self._create_typed_send("laugh"),
MessageType.ANO_SUFFIX: self._create_typed_send("ano_suffix"),
}
for func in self.send_functions.values():
self.backoff_svc.add_callable(func, backoff_seconds=10)
def _create_typed_send(self, type_name: str):
"""Creates a send function for a specific message type"""
async def typed_send(content: str):
if hasattr(self, "current_message"):
await self.current_message.channel.send(content)
# Set a unique name for the function to ensure it's treated as distinct
typed_send.__name__ = f"send_{type_name}"
return typed_send
async def _send_message(self, content: str):
"""Generic send message function (for non-backoff uses)"""
if hasattr(self, "current_message"):
await self.current_message.channel.send(content)
def is_bot_mention(self, tok: str) -> bool:
return tok.lower() == str(self.bot.nick).lower()
async def _handle_bot_mention(self, message: Message) -> str | None:
content = (message.content or "").strip()
if not content:
return None
tokens = content.split()
contains_mention = any(self.is_bot_mention(t) for t in tokens)
if not contains_mention:
return None
# Find the first non-mention token as seed
non_mention_tokens = (
t.strip(".,!?;:") for t in tokens if not self.is_bot_mention(t)
)
seed = next((t for t in non_mention_tokens if t), None)
if not seed:
return None
sentence = await self.generate_svc.run(seed)
if not sentence:
return None
await message.channel.send(f"@{message.author.name} {sentence}")
return sentence
@commands.Cog.event()
async def event_message(self, message):
"""Main message event handler"""
if not message.author:
return
self.current_message = message
await self.store_svc.run(message.content)
# If the message contains a mention to this bot, reply by generating
# a sentence from the first word that is not the bot username itself.
if await self._handle_bot_mention(message):
# If the bot actually replies with something, it should not try to send
# any other type of reply
return
msg_type = self.message_handler.get_message_type(message.content)
response = None
match msg_type:
case MessageType.COMMAND:
return
case MessageType.YES:
response = "Indeed"
case MessageType.WHAT:
response = "WHAT Ramon"
case MessageType.LAUGH:
response = self.message_handler.handle_laugh()
case MessageType.ANO_SUFFIX:
response = (
f"@{message.author.name} {self.message_handler.handle_ano_suffix()}"
)
case MessageType.OTHER:
return
if response and msg_type in self.send_functions:
await self.backoff_svc.call_async(self.send_functions[msg_type], response)
class BotsManager:
def __init__(self, s: Settings):
self.bots: dict[str, Bot] = {}
self.s = s
def add_bot(self, user: User, channel: str, chatbot: Chatbot):
if user.username in self.bots:
logger.info(f"Bot for {user.username} already exists")
return
logger.info(f"Adding bot for {user.username}")
bot = Bot(user=user, channel=channel, chatbot=chatbot, settings=self.s)
bot.add_cog(SaveMessagesCog(bot))
bot.add_cog(HelloMessagesCog(bot))
self.bots[user.username] = bot
async def run_user_bot(self, user: User):
if user.username not in self.bots:
return None
logger.info(f"Starting bot for {user.username}")
bot = self.bots[user.username]
@retry(
stop=stop_after_attempt(5),
wait=wait_exponential(multiplier=2, min=2, max=60),
retry=retry_if_exception_type((ConnectionError, TimeoutError, OSError)),
)
async def start_bot_with_retry():
await bot.start()
task = asyncio.create_task(start_bot_with_retry())
def on_bot_done(future):
try:
if future.cancelled():
logger.warning(f"Bot for {user.username} was cancelled")
elif future.exception():
logger.error(
f"Bot for {user.username} failed: {future.exception()}"
)
else:
logger.info(f"Bot for {user.username} stopped normally")
except Exception as e: # noqa: BLE001
logger.error(f"Error in bot completion callback: {e}")
task.add_done_callback(on_bot_done)
bot.start_routines()
return task
async def run_user_bot2(self, user: User):
if user.username not in self.bots:
return
logger.info(f"Starting bot for {user.username}")
bot = self.bots[user.username]
task = asyncio.create_task(bot.start())
task.add_done_callback(
lambda x: logger.info(f"Bot for {user.username} stopped")
)
bot.start_routines()
async def stop_user_bot(self, user: User):
if user.username not in self.bots:
return
bot = self.bots.pop(user.username)
await bot.close()
bot.stop_routines()

View file

View file

@ -0,0 +1,77 @@
import httpx
from litestar.exceptions import HTTPException
from pydantic import BaseModel, ConfigDict, Field
from huesoporro.models import TwitchAuth
from huesoporro.settings import Settings
class TwitchAuthenticator(BaseModel):
s: Settings = Field(default_factory=Settings.get)
client: httpx.AsyncClient = Field(
default_factory=lambda x: httpx.AsyncClient(base_url="https://id.twitch.tv/")
)
model_config = ConfigDict(arbitrary_types_allowed=True)
async def get_token(self, code: str, auto_refresh: bool = True) -> TwitchAuth:
response = await self.client.post(
"/oauth2/token",
data={
"client_id": self.s.twitch_client_id,
"client_secret": self.s.twitch_client_secret.get_secret_value(),
"grant_type": "authorization_code",
"code": code,
"redirect_uri": f"{self.s.server_hostname}o/code",
},
headers={"Accept": "application/json"},
)
if auto_refresh and response.status_code == 401: # noqa: PLR2004
return await self.refresh_token(response.json()["refresh_token"])
response.raise_for_status()
profile = await self.get_userinfo(response.json()["access_token"])
return TwitchAuth(**response.json(), userinfo=profile)
async def get_userinfo(self, access_token):
response = await self.client.get(
"/oauth2/userinfo", headers={"Authorization": f"Bearer {access_token}"}
)
response.raise_for_status()
return response.json()
async def refresh_token(self, refresh_token: str) -> TwitchAuth:
response = await self.client.post(
"/oauth2/token",
data={
"client_id": Settings.get().twitch_client_id,
"client_secret": Settings.get().twitch_client_secret.get_secret_value(),
"grant_type": "refresh_token",
"refresh_token": refresh_token,
},
headers={"Accept": "application/json"},
)
response.raise_for_status()
profile = await self.get_userinfo(response.json()["access_token"])
return TwitchAuth(**response.json(), userinfo=profile)
async def validate_token(self, access_token: str) -> str:
response = await self.client.get(
"/oauth2/validate", headers={"Authorization": f"OAuth {access_token}"}
)
response.raise_for_status()
user_data = response.json()
if user_data.get("status"):
raise HTTPException(status_code=401, detail="Unauthorized")
if (user := user_data["login"]) not in self.s.allowed_users:
raise HTTPException(status_code=403, detail="Forbidden")
return user
async def token_is_valid(self, access_token: str) -> bool:
response = await self.client.get(
"/oauth2/validate", headers={"Authorization": f"OAuth {access_token}"}
)
return response.status_code == 200 # noqa: PLR2004

View file

@ -0,0 +1,411 @@
import json
from abc import ABC, abstractmethod
from contextlib import asynccontextmanager
from typing import Generic, TypeVar
from uuid import UUID
import aiosqlite
from pydantic import BaseModel, Field
from huesoporro import utils
from huesoporro.models import Chatbot, Quote, User
from huesoporro.settings import Settings
T = TypeVar("T", bound=BaseModel)
class IRepo(BaseModel, ABC, Generic[T]):
s: Settings = Field(default_factory=Settings.get)
@asynccontextmanager
async def get_client(self, auto_commit=True):
async with aiosqlite.connect(self.s.db_filepath) as db:
db.row_factory = aiosqlite.Row
yield db
if auto_commit:
await db.commit()
@abstractmethod
async def create(self, obj: T, auto_commit=True) -> T:
pass # pragma: no cover
@abstractmethod
async def update(self, obj: T, auto_commit=True) -> T:
pass # pragma: no cover
@abstractmethod
async def delete(self, obj: T, auto_commit=True):
pass # pragma: no cover
@abstractmethod
async def get_by_id(self, obj_id: UUID, auto_commit=True) -> T | None:
pass # pragma: no cover
@abstractmethod
async def list(self, offset: int = 0, limit: int = 10, auto_commit=True) -> list[T]:
pass # pragma: no cover
class UserRepo(IRepo[User]):
@staticmethod
def _deserialize(data: dict) -> User:
return User(
id=UUID(data["id"]),
username=data["username"],
created_at=data["created_at"],
last_updated_at=data["last_updated_at"],
external_auth=json.loads(data["external_auth"]),
)
async def get_by_id(self, obj_id: UUID, auto_commit=True) -> User | None:
async with (
self.get_client(auto_commit=auto_commit) as db,
await db.execute(
"""
SELECT *
FROM users
WHERE id = ?
""",
(obj_id.hex,),
) as cursor,
):
data = await cursor.fetchone()
if not data:
return None
return self._deserialize(data)
async def create(self, obj: User, auto_commit=True) -> User:
if await self.get_by_username(obj.username):
raise ValueError(f"User {obj.username} already exists")
async with (
self.get_client(auto_commit=auto_commit) as db,
await db.execute(
"""INSERT INTO users (id, username, external_auth, created_at, last_updated_at)
VALUES (?, ?, ?, ?, ?) RETURNING *
""",
(
obj.id.hex,
obj.username,
obj.serialize_external_auth(),
obj.created_at,
obj.last_updated_at,
),
) as cursor,
):
data = await cursor.fetchone()
return self._deserialize(data)
async def update(self, obj: User, auto_commit=True) -> User:
if not await self.get_by_id(obj.id):
raise ValueError(f"User {obj.username} does not exist")
async with (
self.get_client(auto_commit=auto_commit) as db,
db.execute(
"""
UPDATE users
SET username = ?,
external_auth = ?,
last_updated_at = ?
WHERE id = ? RETURNING *
""",
(
obj.username,
obj.serialize_external_auth(),
obj.last_updated_at,
obj.id.hex,
),
) as cursor,
):
data = await cursor.fetchone()
return self._deserialize(data)
async def delete(self, obj: User, auto_commit=True):
async with self.get_client(auto_commit=auto_commit) as db:
await db.execute(
"""
DELETE
FROM users
WHERE id = ?
""",
(obj.id.hex,),
)
async def get_by_username(self, user: str, auto_commit=True) -> User | None:
async with (
self.get_client(auto_commit=auto_commit) as db,
db.execute(
"""
SELECT *
FROM users
WHERE username = ?
""",
(user,),
) as cursor,
):
data = await cursor.fetchone()
if not data:
return None
return User(
id=UUID(data["id"]),
username=data["username"],
created_at=data["created_at"],
last_updated_at=data["last_updated_at"],
external_auth=json.loads(data["external_auth"]),
)
async def list( # type: ignore[empty-body]
self, offset: int = 0, limit: int = 10, auto_commit=True
) -> list[User]:
pass # pragma: no cover
async def count(self, obj: User, auto_commit=True):
pass # pragma: no cover
class QuoteRepo(IRepo[Quote]):
@staticmethod
def _deserialize(data: dict) -> Quote:
return Quote(
id=UUID(data["id"]),
quote=data["quote"],
author=data["author"],
channel_name=data["channel"],
created_at=data["created_at"],
last_updated_at=data["last_updated_at"],
is_active=data["is_active"],
)
async def create(self, obj: Quote, auto_commit=True) -> Quote:
if await self.get_by_quote(obj.quote):
raise ValueError(f"Quote {obj.quote} already exists")
async with (
self.get_client(auto_commit=auto_commit) as db,
await db.execute(
"""
INSERT INTO quotes (id, quote, author, channel, created_at, is_active, last_updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?) RETURNING *
""",
(
obj.id.hex,
obj.quote,
obj.author,
obj.channel_name,
obj.created_at,
obj.is_active,
obj.last_updated_at,
),
) as cursor,
):
data = await cursor.fetchone()
return self._deserialize(data)
async def update(self, obj: Quote, auto_commit=True) -> Quote: # type: ignore[empty-body]
if not await self.get_by_id(obj.id):
raise ValueError(f"Quote {obj.id} does not exist")
async with (
self.get_client(auto_commit=auto_commit) as db,
await db.execute(
"""
UPDATE quotes
SET quote = ?,
author = ?,
channel = ?,
is_active = ?,
last_updated_at = ?
WHERE id = ? RETURNING *
""",
(
obj.quote,
obj.author,
obj.channel_name,
obj.is_active,
utils.get_utc_now(),
obj.id.hex,
),
) as cursor,
):
data = await cursor.fetchone()
return self._deserialize(data)
async def delete(self, obj: Quote, auto_commit=True):
async with self.get_client(auto_commit=auto_commit) as db:
await db.execute(
"""
DELETE
FROM quotes
WHERE id = ?
""",
(obj.id.hex,),
)
async def get_by_id(self, obj_id: UUID, auto_commit=True) -> Quote | None: # type: ignore[empty-body]
async with (
self.get_client(auto_commit=auto_commit) as db,
db.execute(
"""
SELECT *
FROM quotes
WHERE id = ?
""",
(obj_id.hex,),
) as cursor,
):
data = await cursor.fetchone()
if not data:
return None
return self._deserialize(data)
async def get_by_quote(self, quote: str, auto_commit=True) -> Quote | None:
async with (
self.get_client(auto_commit=auto_commit) as db,
db.execute(
"""
SELECT *
FROM quotes
WHERE quote = ?
""",
(quote,),
) as cursor,
):
data = await cursor.fetchone()
if not data:
return None
return self._deserialize(data)
async def list( # type: ignore[empty-body]
self, offset: int = 0, limit: int = 10, auto_commit=True
) -> list[Quote]:
async with self.get_client() as db:
db.row_factory = aiosqlite.Row
async with db.execute(
"SELECT * FROM quotes LIMIT ? OFFSET ?", (limit, offset)
) as cursor:
results = await cursor.fetchall()
return [self._deserialize(result) for result in results]
async def get_random(self, channel_name: str, auto_commit=True) -> Quote | None:
async with (
self.get_client(auto_commit=auto_commit) as db,
db.execute(
"""
SELECT *
FROM quotes
WHERE channel = ?
AND is_active = 1
ORDER BY RANDOM() LIMIT 1
""",
(channel_name,),
) as cursor,
):
data = await cursor.fetchone()
if not data:
return None
return self._deserialize(data)
class ChatbotRepo(IRepo[Chatbot]):
@staticmethod
def _deserialize(data: dict) -> Chatbot:
return Chatbot(
id=UUID(data["id"]),
user_id=data["user_id"],
automatic_generation_timer=data["automatic_generation_timer"],
automatic_quote_timer=data["automatic_quote_timer"],
mods=data["mods"].split(","),
last_updated_at=data["last_updated_at"],
created_at=data["created_at"],
)
async def create(self, obj: Chatbot, auto_commit=True) -> Chatbot:
if await self.get_by_user_id(obj.user_id):
raise ValueError(f"Chatbot {obj.user_id} already exists")
async with (
self.get_client(auto_commit=auto_commit) as db,
await db.execute(
"""INSERT INTO chatbot (id,
user_id,
automatic_generation_timer,
automatic_quote_timer,
mods,
created_at,
last_updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?) RETURNING *
""",
(
obj.id.hex,
obj.user_id.hex,
obj.automatic_generation_timer,
obj.automatic_quote_timer,
obj.mods_as_string,
obj.created_at,
obj.last_updated_at,
),
) as cursor,
):
data = await cursor.fetchone()
return self._deserialize(data)
async def update(self, obj: Chatbot, auto_commit=True) -> Chatbot:
if not await self.get_by_user_id(obj.user_id):
raise ValueError(f"Chatbot {obj.user_id} does not exist")
async with (
self.get_client(auto_commit=auto_commit) as db,
await db.execute(
"""UPDATE chatbot
SET automatic_generation_timer = ?,
automatic_quote_timer = ?,
mods = ?,
last_updated_at = ?
WHERE user_id = ? RETURNING *
""",
(
obj.automatic_generation_timer,
obj.automatic_quote_timer,
obj.mods_as_string,
utils.get_utc_now(),
obj.user_id.hex,
),
) as cursor,
):
data = await cursor.fetchone()
return self._deserialize(data)
async def delete(self, obj: Chatbot, auto_commit=True):
if not await self.get_by_id(obj.id):
raise ValueError(f"Chatbot {obj.id} does not exist")
async with self.get_client() as db:
await db.execute("DELETE FROM chatbot WHERE id = ?", (obj.id.hex,))
async def get_by_id(self, obj_id: UUID, auto_commit=True) -> Chatbot | None: # type: ignore[empty-body]
async with self.get_client() as db:
db.row_factory = aiosqlite.Row
async with db.execute(
"SELECT * FROM chatbot WHERE id = ?", (obj_id.hex,)
) as cursor:
result = await cursor.fetchone()
if not result:
return None
return self._deserialize(result)
async def get_by_user_id(self, user_id: UUID) -> Chatbot | None:
async with self.get_client() as db:
db.row_factory = aiosqlite.Row
async with db.execute(
"SELECT * FROM chatbot WHERE user_id = ?", (user_id.hex,)
) as cursor:
result = await cursor.fetchone()
if not result:
return None
return self._deserialize(result)
async def list( # type: ignore[empty-body]
self, offset: int = 0, limit: int = 10, auto_commit=True
) -> list[Chatbot]:
async with self.get_client() as db:
db.row_factory = aiosqlite.Row
async with db.execute(
"SELECT * FROM chatbot LIMIT ? OFFSET ?", (limit, offset)
) as cursor:
results = await cursor.fetchall()
return [self._deserialize(result) for result in results]

View file

@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
SOFTWARE.

View file

@ -257,7 +257,7 @@ This bot is controlled by a `settings.json` file, which has the following struct
| `AutomaticGenerationTimer` | The amount of seconds between automatically sending a generated message, as if someone wrote `!g`. -1 for no automatic generations. | `-1` |
| `WhisperCooldown` | Allows the bot to whisper a user the remaining cooldown after that user has attempted to generate a message. | `true` |
| `EnableGenerateCommand` | Globally enables/disables the generate command. | `true` |
| `SentenceSeparator` | The separator between multiple sentences. Only relevant if `MinSentenceWordAmount` > 0, as only then can multiple sentences be generated. Sensible values for this might be `", "`, `". "`, `" - "` or `" "`. | `" - "` |
| `SentenceSeparator` | The separator between multiple sentences. Only relevant if `MinSentenceWordAmount` > 0, as only then can multiple sentences be generated. Sensible values for this might be `", "`, `". "`, `" - "` or `" "`. | `" - "` |
| `AllowGenerateParams` | Allow chat to supply a partial sentence which the bot finishes, e.g. `!generate hello, I am`. If `false`, all values after the generation command will be ignored. | `true` |
| `GenerateCommands` | The generation commands that the bot will listen for. Defaults to `["!generate", "!g"]`. Useful if your chat is used to commands with `~`, `-`, `/`, etc. | `["!generate", "!g"]` |

View file

View file

@ -1,15 +1,15 @@
import sqlite3
import logging
import random
import sqlite3
import string
from typing import Any, List, Optional, Tuple
from typing import Any
import platformdirs
from loguru import logger
logger = logging.getLogger(__name__)
from huesoporro.settings import Settings
class Database:
class MarkovDatabase:
"""
The database created is called `MarkovChain_{channel}.db`,
and populated with 27 + 27^2 = 756 tables. Firstly, 27 tables with the structure of
@ -87,35 +87,34 @@ class Database:
to both get results from "hello" and "hello,".
"""
def __init__(self, channel: str):
self.user_data_path = platformdirs.user_data_path(
"markovbot_gui", ensure_exists=True
)
self.db_path = (
self.user_data_path / f"MarkovChain_{channel.replace('#', '').lower()}.db"
)
self._execute_queue = []
def __init__(self, channel: str, settings: Settings | None = None):
settings = settings or Settings.get()
self.db_path = settings.default_data_path / f"MarkovChain_{channel}.db"
self.user_data_path = self.db_path.parent
self._execute_queue: list = []
if self.db_path.is_file():
# Ensure the database is updated to the newest version
self.update_v1(channel)
self.update_v2()
self.update_v3(channel)
self.update_v4()
# Create database tables.
for first_char in list(string.ascii_uppercase) + ["_"]:
for first_char in [*list(string.ascii_uppercase), "_"]:
self.add_execute_queue(
f"""
CREATE TABLE IF NOT EXISTS MarkovStart{first_char} (
word1 TEXT COLLATE NOCASE,
word2 TEXT COLLATE NOCASE,
count INTEGER,
word1 TEXT COLLATE NOCASE,
word2 TEXT COLLATE NOCASE,
count INTEGER,
PRIMARY KEY (word1 COLLATE BINARY, word2 COLLATE BINARY)
);
""",
auto_commit=False,
)
for second_char in list(string.ascii_uppercase) + ["_"]:
for second_char in [*list(string.ascii_uppercase), "_"]:
self.add_execute_queue(
f"""
CREATE TABLE IF NOT EXISTS MarkovGrammar{first_char}{second_char} (
@ -186,17 +185,19 @@ class Database:
"""
# If an old version of the Database is used, update the database
if ("MarkovGrammarA",) in self.execute(
"SELECT name FROM sqlite_master WHERE type='table';", fetch=True
"SELECT name FROM sqlite_master WHERE type='table';",
fetch=True,
):
logger.info("Creating backup before updating Database...")
# Connect to both the new and backup, backup, and close both
def progress(status, remaining, total):
logging.debug(f"Copied {total-remaining} of {total} pages...")
logging.debug(f"Copied {total - remaining} of {total} pages...")
conn = sqlite3.connect(f"MarkovChain_{channel.replace('#', '').lower()}.db")
back_conn = sqlite3.connect(
f"MarkovChain_{channel.replace('#', '').lower()}_backup.db"
f"MarkovChain_{channel.replace('#', '').lower()}_backup.db",
)
with back_conn:
conn.backup(back_conn, pages=1000, progress=progress)
@ -209,9 +210,9 @@ class Database:
# Rename ...Other to ..._
self.add_execute_queue("""
CREATE TABLE IF NOT EXISTS MarkovStart_ (
word1 TEXT COLLATE NOCASE,
word2 TEXT COLLATE NOCASE,
occurances INTEGER,
word1 TEXT COLLATE NOCASE,
word2 TEXT COLLATE NOCASE,
occurances INTEGER,
PRIMARY KEY (word1 COLLATE BINARY, word2 COLLATE BINARY)
);
""")
@ -228,10 +229,10 @@ class Database:
# Copy data from Other to _ and remove Other
self.add_execute_queue(
"INSERT INTO MarkovGrammar_ SELECT * FROM MarkovGrammarOther;"
"INSERT INTO MarkovGrammar_ SELECT * FROM MarkovGrammarOther;",
)
self.add_execute_queue(
"INSERT INTO MarkovStart_ SELECT * FROM MarkovStartOther;"
"INSERT INTO MarkovStart_ SELECT * FROM MarkovStartOther;",
)
self.add_execute_queue("DROP TABLE MarkovGrammarOther")
self.add_execute_queue("DROP TABLE MarkovStartOther")
@ -241,17 +242,17 @@ class Database:
# Same with MarkovStart.
for character in list(string.digits):
self.add_execute_queue(
f"INSERT INTO MarkovGrammar_ SELECT * FROM MarkovGrammar{character}"
f"INSERT INTO MarkovGrammar_ SELECT * FROM MarkovGrammar{character}", # noqa: S608
)
self.add_execute_queue(f"DROP TABLE MarkovGrammar{character}")
self.add_execute_queue(
f"INSERT INTO MarkovStart_ SELECT * FROM MarkovStart{character}"
f"INSERT INTO MarkovStart_ SELECT * FROM MarkovStart{character}", # noqa: S608
)
self.add_execute_queue(f"DROP TABLE MarkovStart{character}")
self.execute_commit()
# Split up MarkovGrammarA into MarkovGrammarAA, MarkovGrammarAB, etc.
for first_char in list(string.ascii_uppercase) + ["_"]:
for first_char in [*list(string.ascii_uppercase), "_"]:
for second_char in list(string.ascii_uppercase):
self.add_execute_queue(f"""
CREATE TABLE IF NOT EXISTS MarkovGrammar{first_char}{second_char} (
@ -263,10 +264,10 @@ class Database:
);
""")
self.add_execute_queue(
f'INSERT INTO MarkovGrammar{first_char}{second_char} SELECT * FROM MarkovGrammar{first_char} WHERE word2 LIKE "{second_char}%";'
f'INSERT INTO MarkovGrammar{first_char}{second_char} SELECT * FROM MarkovGrammar{first_char} WHERE word2 LIKE "{second_char}%";', # noqa: S608
)
self.add_execute_queue(
f'DELETE FROM MarkovGrammar{first_char} WHERE word2 LIKE "{second_char}%";'
f'DELETE FROM MarkovGrammar{first_char} WHERE word2 LIKE "{second_char}%";', # noqa: S608
)
self.add_execute_queue(f"""
@ -279,7 +280,7 @@ class Database:
);
""")
self.add_execute_queue(
f"INSERT INTO MarkovGrammar{first_char}_ SELECT * FROM MarkovGrammar{first_char};"
f"INSERT INTO MarkovGrammar{first_char}_ SELECT * FROM MarkovGrammar{first_char};", # noqa: S608
)
self.add_execute_queue(f"DROP TABLE MarkovGrammar{first_char}")
self.execute_commit()
@ -300,17 +301,17 @@ class Database:
fetch=True,
):
logger.info("Updating Database to new version...")
for first_char in list(string.ascii_uppercase) + ["_"]:
for second_char in list(string.ascii_uppercase) + ["_"]:
for first_char in [*list(string.ascii_uppercase), "_"]:
for second_char in [*list(string.ascii_uppercase), "_"]:
self.execute(
f"ALTER TABLE MarkovGrammar{first_char}{second_char} RENAME COLUMN occurances TO count;"
f"ALTER TABLE MarkovGrammar{first_char}{second_char} RENAME COLUMN occurances TO count;",
)
self.execute(
f"ALTER TABLE MarkovStart{first_char} RENAME COLUMN occurances TO count;"
f"ALTER TABLE MarkovStart{first_char} RENAME COLUMN occurances TO count;",
)
logger.info("Finished Updating Database to new version.")
def update_v3(self, channel: str) -> None:
def update_v3(self, channel: str) -> None: # noqa: C901, PLR0915
"""Update the Database structure to mark punctuation as a separate word.
Previously, "Hello," was a valid single word. Now, it would be split as "Hello" and ",".
@ -338,26 +339,31 @@ class Database:
# in which case we definitely want to upgrade.
try:
version = self.execute(
"SELECT version FROM Version ORDER BY version DESC LIMIT 1;", fetch=True
"SELECT version FROM Version ORDER BY version DESC LIMIT 1;",
fetch=True,
)
except sqlite3.OperationalError:
version = []
# Whether to upgrade
if not version or version[0][0] < 3:
if not version or version[0][0] < 3: # noqa: PLR2004
logger.info(
"Updating Database to new version - supports better punctuation handling."
"Updating Database to new version - supports better punctuation handling.",
)
from shutil import copyfile
import os
from Tokenizer import tokenize
from nltk import ngrams
from huesoporro.libs.tokenizer import tokenize
channel = channel.replace("#", "").lower()
copyfile(f"MarkovChain_{channel}.db", f"MarkovChain_{channel}_modified.db")
copyfile(
self.db_path,
self.user_data_path / f"MarkovChain_{channel}_modified.db",
)
logger.info(
f'Created a copy of the database called "MarkovChain_{channel}_modified.db". The update will modify this file.'
f'Created a copy of the database called "MarkovChain_{channel}_modified.db". The update will modify this file.',
)
# Temporarily set self.db_name to the modified one
@ -367,20 +373,20 @@ class Database:
)
# Create database tables.
for first_char in list(string.ascii_uppercase) + ["_"]:
for first_char in [*list(string.ascii_uppercase), "_"]:
table = f"MarkovStart{first_char}"
self.add_execute_queue(
f"""
CREATE TABLE IF NOT EXISTS {table}_modified (
word1 TEXT COLLATE NOCASE,
word2 TEXT COLLATE NOCASE,
count INTEGER,
word1 TEXT COLLATE NOCASE,
word2 TEXT COLLATE NOCASE,
count INTEGER,
PRIMARY KEY (word1 COLLATE BINARY, word2 COLLATE BINARY)
);
""",
auto_commit=False,
)
for second_char in list(string.ascii_uppercase) + ["_"]:
for second_char in [*list(string.ascii_uppercase), "_"]:
table = f"MarkovGrammar{first_char}{second_char}"
self.add_execute_queue(
f"""
@ -396,23 +402,23 @@ class Database:
)
self.execute_commit()
def modify_start(table: str) -> None:
"""Read all data from `table`, re-tokenize it, distribute the new first 2 tokens to _modified tables, and drop `table`.
def modify_start(table_name: str) -> None:
"""Read all data from `table_name`, re-tokenize it, distribute the new first 2 tokens to _modified tables, and drop `table`.
Args:
table (str): The name of the table to work on.
table_name (str): The name of the table to work on.
"""
data = self.execute(f"SELECT * FROM {table};", fetch=True)
data = self.execute(f"SELECT * FROM {table_name};", fetch=True) # noqa: S608
for tup in data:
# Remove "count" from tup for now
count = tup[-1]
tup = tup[:-1]
tup = tup[:-1] # noqa: PLW2901
raw_string = " ".join(tup)
tokenized = tokenize(raw_string)
two_gram = tokenized[:2]
# In case there was some issue in the previous Database
if len(two_gram) < 2:
if len(two_gram) < 2: # noqa: PLR2004
continue
self.add_execute_queue(
f"""
@ -425,31 +431,31 @@ class Database:
),
1
)
)""",
)""", # noqa: S608
values=two_gram + two_gram,
auto_commit=False,
)
self.execute(f"DROP TABLE {table};")
self.execute(f"DROP TABLE {table_name};")
def modify_grammar(table: str) -> None:
"""Read all data from `table`, re-tokenize it, distribute the new 3-grams to _modified tables, and drop `table`.
def modify_grammar(table_name: str) -> None:
"""Read all data from `table_name`, re-tokenize it, distribute the new 3-grams to _modified tables, and drop `table`.
Args:
table (str): The name of the table to work on.
table_name (str): The name of the table to work on.
"""
data = self.execute(f"SELECT * FROM {table};", fetch=True)
data = self.execute(f"SELECT * FROM {table_name};", fetch=True) # noqa: S608
for tup in data:
# Remove "count" from tup for now
count = tup[-1]
tup = tup[:-1]
tup = tup[:-1] # noqa: PLW2901
# If ends on "<END>", ignore that in in the tuple, as we don't want it to get
# tokenized.
end = False
if tup[-1] == "<END>":
end = True
tup = tup[:-1]
tup = tup[:-1] # noqa: PLW2901
raw_string = " ".join(tup)
tokenized = tokenize(raw_string)
@ -474,39 +480,39 @@ class Database:
),
1
)
)""",
)""", # noqa: S608
values=ngram + ngram,
auto_commit=False,
)
self.execute(f"DROP TABLE {table};")
self.execute(f"DROP TABLE {table_name};")
# Modify all tables
i = 0
total = 27 * 27 + 27 # The number of tables to convert
for first_char in list(string.ascii_uppercase) + ["_"]:
for first_char in [*list(string.ascii_uppercase), "_"]:
table = f"MarkovStart{first_char}"
modify_start(table)
i += 1
for second_char in list(string.ascii_uppercase) + ["_"]:
for second_char in [*list(string.ascii_uppercase), "_"]:
table = f"MarkovGrammar{first_char}{second_char}"
modify_grammar(table)
i += 1
logger.debug(
f"[{i / total * 100:.2f}%] Scheduled updates for the tables for words starting in {first_char}."
f"[{i / total * 100:.2f}%] Scheduled updates for the tables for words starting in {first_char}.",
)
logger.info("Starting executing table update...")
self.execute_commit()
logger.info("Finished executing table update.")
# Rename the _modified tables to normal tables again
for first_char in list(string.ascii_uppercase) + ["_"]:
for first_char in [*list(string.ascii_uppercase), "_"]:
table = f"MarkovStart{first_char}"
self.add_execute_queue(
f"ALTER TABLE {table}_modified RENAME TO {table};",
auto_commit=False,
)
for second_char in list(string.ascii_uppercase) + ["_"]:
for second_char in [*list(string.ascii_uppercase), "_"]:
table = f"MarkovGrammar{first_char}{second_char}"
self.add_execute_queue(
f"ALTER TABLE {table}_modified RENAME TO {table};",
@ -516,8 +522,12 @@ class Database:
# Turn the non-modified, old version of the Database into a "_backup.db" file,
# and turn the modified file into the new main file.
os.rename(f"MarkovChain_{channel}.db", f"MarkovChain_{channel}_backup.db")
os.rename(f"MarkovChain_{channel}_modified.db", f"MarkovChain_{channel}.db")
self.db_path.rename(self.db_path / f"MarkovChain_{channel}_backup.db")
(self.user_data_path / f"MarkovChain_{channel}_modified.db").rename(
self.db_path / f"MarkovChain_{channel}.db",
)
# os.rename(f"MarkovChain_{channel}.db", f"MarkovChain_{channel}_backup.db") # noqa: ERA001
# os.rename(f"MarkovChain_{channel}_modified.db", f"MarkovChain_{channel}.db") # noqa: ERA001
# Revert to using .db instead of _modified.db
self.db_path = (
@ -533,17 +543,42 @@ class Database:
self.execute("INSERT INTO Version (version) VALUES (3);")
logger.info(
f'Renamed original database file "MarkovChain_{channel}.db" to "MarkovChain_{channel}_backup.db". This file is *not* used, and can safely be deleted.'
f'Renamed original database file "MarkovChain_{channel}.db" to "MarkovChain_{channel}_backup.db". This file is *not* used, and can safely be deleted.',
)
logger.info(
f'Renamed updated database file "MarkovChain_{channel}_modified.db" to "MarkovChain_{channel}.db".'
f'Renamed updated database file "MarkovChain_{channel}_modified.db" to "MarkovChain_{channel}.db".',
)
logger.info(
f'This updated "MarkovChain_{channel}.db" will be used to drive the Twitch bot.'
f'This updated "MarkovChain_{channel}.db" will be used to drive the Twitch bot.',
)
def update_v4(self):
"""Update the db schema to allow storing quotes in a new `quotes` table. Besides the id, the table stores the
quote and the author. The quote is unique.
"""
# Get Database version. Throws OperationalError if the Version table does not exist,
# in which case we definitely want to upgrade.
try:
version = self.execute(
"SELECT version FROM Version ORDER BY version DESC LIMIT 1;",
fetch=True,
)
except sqlite3.OperationalError:
version = []
if not version or version[0][0] < 4: # noqa: PLR2004
logger.info("Updating db to the v4 version")
self.execute("""CREATE TABLE IF NOT EXISTS quotes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
quote TEXT UNIQUE,
author TEXT
);""")
def add_execute_queue(
self, sql: str, values: Tuple[Any] = None, auto_commit: bool = True
self,
sql: str,
values: tuple[Any] | list[Any] | None = None,
auto_commit: bool = True,
) -> None:
"""Add query and corresponding values to a queue, to be executed all at once.
@ -553,16 +588,21 @@ class Database:
Args:
sql (str): The SQL query to add, potentially with "?" for where
a value ought to be filled in.
values ([Tuple[Any]], optional): Optional tuple of values to replace "?" in SQL queries.
values ([tuple[Any]], optional): Optional tuple of values to replace "?" in SQL queries.
Defaults to None.
"""
if values is not None:
self._execute_queue.append([sql, values])
self._execute_queue.append((sql, values))
else:
self._execute_queue.append([sql])
# Commit these executes if there are more than 25 queries
if auto_commit and len(self._execute_queue) > 25:
self._execute_queue.append((sql,))
# Commit these executes if there are more than 5 queries
if auto_commit and len(self._execute_queue) > 4: # noqa: PLR2004
logger.info(f"Queue length limit reached, executing query {sql}")
self.execute_commit()
else:
logger.info(
f"Not enough queries in queue to commit: {len(self._execute_queue)}/5"
)
def execute_commit(self, fetch: bool = False) -> Any:
"""Execute the SQL queries added to the queue with `self.add_execute_queue`.
@ -581,17 +621,18 @@ class Database:
for sql in self._execute_queue:
cur.execute(*sql)
self._execute_queue.clear()
cur.execute("commit")
conn.commit()
if fetch:
return cur.fetchall()
return None
def execute(self, sql: str, values: Tuple[Any] = None, fetch: bool = False):
def execute(self, sql: str, values: tuple[Any] | None = None, fetch: bool = False):
"""Execute the SQL query with the corresponding values, potentially returning a result.
Args:
sql (str): The SQL query to add, potentially with "?" for where
a value ought to be filled in.
values ([Tuple[Any]], optional): Optional tuple of values to replace "?" in SQL queries.
values ([tuple[Any]], optional): Optional tuple of values to replace "?" in SQL queries.
Defaults to None.
fetch (bool, optional): Whether to return the fetchall() of the SQL queries.
Defaults to False.
@ -608,6 +649,7 @@ class Database:
conn.commit()
if fetch:
return cur.fetchall()
return None
@staticmethod
def get_suffix(character: str) -> str:
@ -636,7 +678,7 @@ class Database:
values=(username,),
)
def check_whisper_ignore(self, username: str) -> List[Tuple[str]]:
def check_whisper_ignore(self, username: str) -> list[tuple[str]]:
"""Returns a non-empty list only if `username` is in the WhisperIgnore table.
Otherwise, returns an empty list. Is used to ensure that a user who doesn't want to be
@ -646,7 +688,7 @@ class Database:
username (str): The username of the user to check.
Returns:
List[Tuple[str]]: Either an empty list, or [('test_user',)].
list[tuple[str]]: Either an empty list, or [('test_user',)].
Allows the use of `if not check_whisper_ignore(user): whisper(user)`
"""
return self.execute(
@ -685,14 +727,14 @@ class Database:
"""
return items[0] * len(items) == items
def get_next(self, index: int, words: List[str]) -> Optional[str]:
def get_next(self, index: int, words: list | tuple) -> str | None:
"""Generate the next word in the sentence using learned data, given the previous `key_length` words.
`key_length` is set to 2 by default, and cannot easily be changed.
Args:
index (int): The index of this new word in the sentence.
words (List[str]): The previous 2 words.
words (list[str]): The previous 2 words.
Returns:
Optional[str]: The next word in the sentence, generated given the learned data.
@ -701,14 +743,14 @@ class Database:
data = self.execute(
f"""
SELECT word3, count FROM MarkovGrammar{self.get_suffix(words[0][0])}{self.get_suffix(words[1][0])}
WHERE word1 = ? AND word2 = ?;""",
values=words,
WHERE word1 = ? AND word2 = ?;""", # noqa: S608
values=words, # type: ignore[arg-type]
fetch=True,
)
# Return a word picked from the data, using count as a weighting factor
return None if len(data) == 0 else self.pick_word(data, index)
def get_next_initial(self, index: int, words) -> Optional[str]:
def get_next_initial(self, index: int, words) -> str | None:
"""Generate the next word in the sentence using learned data, given the previous `key_length` words.
`key_length` is set to 2 by default, and cannot easily be changed.
@ -716,7 +758,7 @@ class Database:
Args:
index (int): The index of this new word in the sentence.
words (List[str]): The previous 2 words.
words (list[str]): The previous 2 words.
Returns:
Optional[str]: The next word in the sentence, generated given the learned data.
@ -725,14 +767,14 @@ class Database:
data = self.execute(
f"""
SELECT word3, count FROM MarkovGrammar{self.get_suffix(words[0][0])}{self.get_suffix(words[1][0])}
WHERE word1 = ? AND word2 = ? AND word3 != '<END>';""",
WHERE word1 = ? AND word2 = ? AND word3 != '<END>';""", # noqa: S608
values=words,
fetch=True,
)
# Return a word picked from the data, using count as a weighting factor
return None if len(data) == 0 else self.pick_word(data, index)
def get_next_single_initial(self, index: int, word: str) -> Optional[List[str]]:
def get_next_single_initial(self, index: int, word: str) -> list[str] | None:
"""Generate the next word in the sentence using learned data, given the previous word.
Randomly picks a start character for the second word by weighing all uppercase letters and "_" with their word frequency.
@ -742,47 +784,48 @@ class Database:
word (str): The previous word.
Returns:
Optional[List[str]]: The previous and newly generated word in the sentence as a list, generated given the learned data.
Optional[list[str]]: The previous and newly generated word in the sentence as a list, generated given the learned data.
So, the previous word is taken directly the input of this method, and the second word is generated.
"""
# Randomly pick first character for the second word
char_two = random.choices(
string.ascii_uppercase + "_", weights=self.word_frequency
char_two = random.choices( # noqa: S311
string.ascii_uppercase + "_",
weights=self.word_frequency,
)[0]
# Get all items
data = self.execute(
f"""
SELECT word2, count FROM MarkovGrammar{self.get_suffix(word[0])}{char_two}
WHERE word1 = ? AND word2 != '<END>';""",
WHERE word1 = ? AND word2 != '<END>';""", # noqa: S608
values=(word,),
fetch=True,
)
# Return a word picked from the data, using count as a weighting factor
return None if len(data) == 0 else [word] + [self.pick_word(data, index)]
return None if len(data) == 0 else [word, self.pick_word(data, index)]
def get_next_single_start(self, word: str) -> Optional[List[str]]:
def get_next_single_start(self, word: str) -> list[str] | None:
"""Generate the second word in the sentence using learned data, given the very first word in the sentence.
Args:
word (str): The first word in the sentence.
Returns:
Optional[List[str]]: The first and second word in the sentence as a list, generated given the learned data.
Optional[list[str]]: The first and second word in the sentence as a list, generated given the learned data.
So, the first word is taken directly the input of this method, and the second word is generated.
"""
# Get all items
data = self.execute(
f"""
SELECT word2, count FROM MarkovStart{self.get_suffix(word[0])}
WHERE word1 = ?;""",
WHERE word1 = ?;""", # noqa: S608
values=(word,),
fetch=True,
)
# Return a word picked from the data, using count as a weighting factor
return None if len(data) == 0 else [word] + [self.pick_word(data)]
return None if len(data) == 0 else [word, self.pick_word(data)]
@staticmethod
def pick_word(data: List[Tuple[str, int]], index: int = 0) -> str:
def pick_word(data: list[tuple[str, int]], index: int = 0) -> str:
"""Randomly pick a word from `data` with word frequency as the weight.
`index` is further used to decrease the weight of the <END> token for the first 15 words
@ -797,7 +840,7 @@ class Database:
Returns:
str: The pseudo-randomly picked word.
"""
return random.choices(
return random.choices( # noqa: S311
data,
weights=[
tup[-1] * ((index + 1) / 15) if tup[0] == "<END>" else tup[-1]
@ -805,22 +848,24 @@ class Database:
],
)[0][0]
def get_start(self) -> List[str]:
def get_start(self) -> list[str]:
"""Get a list of two words that mark as the start of a sentence.
This is randomly gathered from MarkovStart{character}.
Returns:
List[str]: A list of two starting words, such as ["I", "am"].
list[str]: A list of two starting words, such as ["I", "am"].
"""
# Find one character start from
character = random.choices(
list(string.ascii_lowercase) + ["_"], weights=self.word_frequency, k=1
character = random.choices( # noqa: S311
[*list(string.ascii_lowercase), "_"],
weights=self.word_frequency,
k=1,
)[0]
# Get all first word, second word, frequency triples,
# e.g. [("I", "am", 3), ("You", "are", 2), ...]
data = self.execute(f"SELECT * FROM MarkovStart{character};", fetch=True)
data = self.execute(f"SELECT * FROM MarkovStart{character};", fetch=True) # noqa: S608
# If nothing has ever been said
if len(data) == 0:
@ -828,10 +873,10 @@ class Database:
# Return a (weighted) randomly chosen 2-gram
return list(
random.choices(data, weights=[tup[-1] for tup in data], k=1)[0][:-1]
random.choices(data, weights=[tup[-1] for tup in data], k=1)[0][:-1], # noqa: S311
)
def add_rule_queue(self, item: List[str]) -> None:
def add_rule_queue(self, item: list[str]) -> None:
"""Adds a rule to the queue, ready to be entered into the knowledge base, given a 3-gram `item`.
The rules on the queue are added with `self.add_execute_queue`,
@ -841,7 +886,7 @@ class Database:
we perform no learning. If we did, this could cause infinite recursion in generation.
Args:
item (List[str]): A 3-gram, e.g. ['How', 'are', 'you']. This is learned by placing this
item (list[str]): A 3-gram, e.g. ['How', 'are', 'you']. This is learned by placing this
in the MarkovGrammarHA table, where it can be seen as:
*Given ["How", "are"], then "you" is a potential output*
The frequency of this word as an output is then incremented,
@ -854,7 +899,7 @@ class Database:
"" in item
): # prevent adding invalid rules. Ideally this wouldn't trigger, but it seems to happen rarely.
logger.warning(
f"Failed to add item to rules. Item contains empty string: {item!r}"
f"Failed to add item to rules. Item contains empty string: {item!r}",
)
return
self.add_execute_queue(
@ -866,18 +911,18 @@ class Database:
WHERE word1 = ? COLLATE BINARY AND word2 = ? COLLATE BINARY AND word3 = ? COLLATE BINARY
),
1)
)""",
)""", # noqa: S608
values=item + item,
)
def add_start_queue(self, item: List[str]) -> None:
def add_start_queue(self, item: list[str]) -> None:
"""Adds a rule to the queue, ready to be entered into the knowledge base, given a 2-gram `item`.
The rules on the queue are added with `self.add_execute_queue`,
which automatically executes the queries in the queue when there are enough queries waiting.
Args:
item (List[str]): A 2-gram, e.g. ['How', 'are']. This is learned by placing this
item (list[str]): A 2-gram, e.g. ['How', 'are']. This is learned by placing this
in the MarkovStartH table, where it can be randomly (with frequency as weight)
picked as a start of a sentence.
"""
@ -890,77 +935,6 @@ class Database:
WHERE word1 = ? COLLATE BINARY AND word2 = ? COLLATE BINARY
),
1)
)""",
)""", # noqa: S608
values=item + item,
)
def unlearn(self, message: str) -> None:
"""Remove frequency of 3-grams from `message` from the knowledge base.
Useful when a message is deleted - usually we want the bot to say those things less frequently.
The frequency count for each of the 3-grams is reduced by 5, i.e. the message is unlearned by 5
times the rate that a message is learned.
If this means the frequency for the 3-gram becomes negative,
we delete the 3-gram from the knowledge base entirely.
Args:
message (str): The message to unlearn.
"""
words = message.split(" ")
# Construct 3-grams
tuples = [
(words[i], words[i + 1], words[i + 2]) for i in range(0, len(words) - 2)
]
# Unlearn start of sentence from MarkovStart
if len(words) > 1:
# Reduce "count" by 5
self.add_execute_queue(
f"""
UPDATE MarkovStart{self.get_suffix(words[0][0])}
SET count = count - 5
WHERE word1 = ? AND word2 = ?;""",
values=(
words[0],
words[1],
),
)
# Delete if count is now less than 0.
self.add_execute_queue(
f"""
DELETE FROM MarkovStart{self.get_suffix(words[0][0])}
WHERE word1 = ? AND word2 = ? AND count <= 0;""",
values=(
words[0],
words[1],
),
)
# Unlearn all 3 word sections from Grammar
for word1, word2, word3 in tuples:
# Reduce "count" by 5
self.add_execute_queue(
f"""
UPDATE MarkovGrammar{self.get_suffix(word1[0])}{self.get_suffix(word2[0])}
SET count = count - 5
WHERE word1 = ? AND word2 = ? AND word3 = ?;""",
values=(
word1,
word2,
word3,
),
)
# Delete if count is now less than 0.
self.add_execute_queue(
f"""
DELETE FROM MarkovGrammar{self.get_suffix(word1[0])}{self.get_suffix(word2[0])}
WHERE word1 = ? AND word2 = ? AND word3 = ? AND count <= 0;""",
values=(
word1,
word2,
word3,
),
)
self.execute_commit()

View file

@ -1,6 +1,6 @@
import threading
import logging
from typing import Callable
import threading
from collections.abc import Callable
logger = logging.getLogger(__name__)
@ -12,7 +12,11 @@ class LoopingTimer(threading.Thread):
"""
def __init__(
self, interval: int, target: Callable[[], None], *args, **kwargs
self,
interval: int,
target: Callable[[], None],
*args,
**kwargs,
) -> None:
threading.Thread.__init__(self)
self.interval = interval

View file

@ -1,26 +1,29 @@
import re
from typing import List
from typing import ClassVar
from nltk.tokenize.destructive import NLTKWordTokenizer
from nltk.tokenize.treebank import TreebankWordDetokenizer
class MarkovChainTokenizer(NLTKWordTokenizer):
# Starting quotes.
STARTING_QUOTES = [
(re.compile("([«“‘„]|[`]+)", re.U), r" \1 "),
# (re.compile(r"^\""), r"``"), # Custom for MarkovChain: Don't use `` as starting quotes
STARTING_QUOTES: ClassVar[list] = [
(re.compile("([«“‘„]|[`]+)", re.UNICODE), r" \1 "), # noqa: RUF001
(re.compile(r"(``)"), r" \1 "),
(re.compile(r"([ \(\[{<])(\"|\'{2})"), r"\1 '' "),
(re.compile(r"(?i)(\')(?!re|ve|ll|m|t|s|d)(\w)\b", re.U), r"\1 \2"),
(re.compile(r"(?i)(\')(?!re|ve|ll|m|t|s|d)(\w)\b", re.UNICODE), r"\1 \2"),
]
PUNCTUATION = [
(re.compile(r""), r"'"),
(re.compile(r'([^\.])(\.)([\]\)}>"\'' "»”’ " r"]*)\s*$", re.U), r"\1 \2 \3 "),
PUNCTUATION: ClassVar[list] = [
(re.compile(r""), r"'"), # noqa: RUF001
(
re.compile(r'([^\.])(\.)([\]\)}>"\'' "»”’ " r"]*)\s*$", re.UNICODE), # noqa: RUF001
r"\1 \2 \3 ",
),
(re.compile(r"([:,])([^\d])"), r" \1 \2"),
(re.compile(r"([:,])$"), r" \1 "),
# See https://github.com/nltk/nltk/pull/2322
(re.compile(r"\.{2,}", re.U), r" \g<0> "),
(re.compile(r"\.{2,}", re.UNICODE), r" \g<0> "),
# Custom for MarkovChain: Removed the "@"
(re.compile(r"[;#$%&]"), r" \g<0> "),
(
@ -30,7 +33,7 @@ class MarkovChainTokenizer(NLTKWordTokenizer):
(re.compile(r"[?!]"), r" \g<0> "),
(re.compile(r"([^'])' "), r"\1 ' "),
# See https://github.com/nltk/nltk/pull/2322
(re.compile(r"[*]", re.U), r" \g<0> "),
(re.compile(r"[*]", re.UNICODE), r" \g<0> "),
]
@ -49,14 +52,14 @@ EMOTICON_RE = re.compile(
|
<3 # heart
)""",
re.VERBOSE | re.I | re.UNICODE,
re.VERBOSE | re.IGNORECASE | re.UNICODE,
)
_tokenize = MarkovChainTokenizer().tokenize
_detokenize = TreebankWordDetokenizer().tokenize
def tokenize(sentence: str) -> List[str]:
def tokenize(sentence: str) -> list[str]:
"""Word tokenize, separating commas, dots, apostrophes, etc.
Uses nltk's `NLTKWordTokenizer`, but does not consider "@" to be punctuation.
@ -68,7 +71,7 @@ def tokenize(sentence: str) -> List[str]:
sentence (str): Input sentence.
Returns:
List[str]: Tokenized output of the sentence.
list[str]: Tokenized output of the sentence.
"""
output = []
@ -85,7 +88,7 @@ def tokenize(sentence: str) -> List[str]:
return output
def detokenize(tokenized: List[str]) -> str:
def detokenize(tokenized: list[str]) -> str:
"""Detokenize a tokenized list of words and punctuation.
Converted in a less naïve way than `" ".join(tokenized)`
@ -107,7 +110,7 @@ def detokenize(tokenized: List[str]) -> str:
index for index, token in enumerate(tokenized) if token in ("''", "'", '"')
]
# Replace '' with ", works better with more recent NLTK versions
tokenized_copy = [token if token != "''" else '"' for token in tokenized]
tokenized_copy = [token if token != "''" else '"' for token in tokenized] # noqa: S105
# We get the reverse of the enumerate, as we modify the list we took the indices from
enumerated = list(enumerate(indices))
@ -117,15 +120,13 @@ def detokenize(tokenized: List[str]) -> str:
# If there is another word, merge with that word and prepend a space
if len(tokenized) > index + 1:
tokenized_copy[index : index + 2] = [
"".join(tokenized_copy[index : index + 2])
"".join(tokenized_copy[index : index + 2]),
]
# Closing quote
else:
# If there is a previous word, merge with that word and append a space
if index > 0:
tokenized_copy[index - 1 : index + 1] = [
"".join(tokenized_copy[index - 1 : index + 1])
]
elif index > 0:
tokenized_copy[index - 1 : index + 1] = [
"".join(tokenized_copy[index - 1 : index + 1]),
]
return _detokenize(tokenized_copy).strip()

7
src/huesoporro/main.py Normal file
View file

@ -0,0 +1,7 @@
import uvicorn
from huesoporro.settings import Settings
if __name__ == "__main__":
settings = Settings.get()
uvicorn.run("src.huesoporro.api.main:app", host=settings.host, port=settings.port)

116
src/huesoporro/models.py Normal file
View file

@ -0,0 +1,116 @@
import datetime
import json
from typing import Literal
import jwt
from pydantic import UUID4, AwareDatetime, BaseModel, Field, field_validator
from huesoporro import utils
from huesoporro.settings import Settings
class TwitchAuth(BaseModel):
access_token: str
refresh_token: str
userinfo: dict
class ExternalAuth(BaseModel):
credentials: dict
type: Literal["twitch"] = "twitch"
class User(BaseModel):
id: UUID4
username: str
external_auth: dict[Literal["twitch", "discord"], TwitchAuth]
created_at: AwareDatetime = Field(default_factory=utils.get_utc_now)
last_updated_at: AwareDatetime = Field(default_factory=utils.get_utc_now)
def encode(
self, settings: Settings | None = None, exclude_fields: set[str] | None = None
) -> str:
s = settings or Settings.get()
exclude_fields = exclude_fields or {"external_auth"}
return jwt.encode(
self.model_dump(exclude=exclude_fields, mode="json"),
key=s.jwt_secret.get_secret_value(),
algorithm="HS256",
)
@classmethod
def decode(cls, token: str, settings: Settings | None = None) -> dict:
s = settings or Settings.get()
return jwt.decode(
token, key=s.jwt_secret.get_secret_value(), algorithms=["HS256"]
)
@property
def twitch_access_token(self):
return self.external_auth["twitch"].access_token
@property
def twitch_refresh_token(self):
return self.external_auth["twitch"].refresh_token
@twitch_access_token.setter # type: ignore[attr-defined,no-redef]
def twitch_access_token(self, value):
self.external_auth["twitch"].access_token = value
@twitch_refresh_token.setter # type: ignore[attr-defined,no-redef]
def twitch_refresh_token(self, value):
self.external_auth["twitch"].refresh_token = value
def serialize_external_auth(self) -> str:
"""Return a JSON string with the inner pydantic model of external_auth serialized using model_dump"""
return json.dumps({k: v.model_dump() for k, v in self.external_auth.items()})
class Chatbot(BaseModel):
"""A chatbot is an entity that holds settings for a given user, it is NOT tied to a channel.
Attributes:
id (UUID4): The unique identifier for the chatbot.
user_id (UUID): The user_id of the user that owns the chatbot.
automatic_generation_timer (int): The timer for automatic generation of quotes.
automatic_quote_timer (int): The timer for automatic quotes.
mods (list[str]): The list of mods for the chatbot.
"""
id: UUID4
user_id: UUID4
automatic_generation_timer: int = 300
automatic_quote_timer: int = 500
created_at: AwareDatetime = Field(default_factory=utils.get_utc_now)
last_updated_at: AwareDatetime = Field(default_factory=utils.get_utc_now)
mods: list[str] = Field(default_factory=list)
@property
def mods_as_string(self):
if not self.mods:
return ""
return ",".join(self.mods)
@field_validator("mods", mode="before")
@classmethod
def format_mods_from_string(cls, v):
if isinstance(v, str):
return v.split(",")
return v
class Quote(BaseModel):
id: UUID4
quote: str
author: str
channel_name: str
is_active: bool = True
created_at: datetime.datetime = Field(default_factory=utils.get_utc_now)
last_updated_at: datetime.datetime = Field(default_factory=utils.get_utc_now)
def as_pretty(self) -> str:
return f"«{self.quote}» - {self.author}"
def as_pretty_saved(self):
return f"He añadido la cita «{self.quote}» de {self.author}"

View file

@ -0,0 +1,52 @@
from functools import lru_cache
from pathlib import Path
import platformdirs
from pydantic import Field, HttpUrl, SecretStr, field_validator
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
port: int = 8000
host: str = "0.0.0.0" # noqa: S104
default_data_path: Path = platformdirs.user_data_path(
"huesoporro",
ensure_exists=True,
)
static_files_path: Path = Field(
default_factory=lambda: Path(__file__).parent / "static"
)
templates_files_path: Path = Field(
default_factory=lambda: Path(__file__).parent / "templates"
)
tts_cache_path: Path = default_data_path / "tts_files"
db_filepath: Path = default_data_path / "huesoporro.db"
twitch_client_id: str
twitch_client_secret: SecretStr
jwt_secret: SecretStr
twitch_scopes: list[str] = Field(
default_factory=lambda: ["channel:bot", "chat:edit", "chat:read"]
)
allowed_users: list[str] | str = Field(default_factory=lambda: ["huesoporro"])
server_hostname: HttpUrl = "http://localhost:8000" # type: ignore[assignment]
@staticmethod
@lru_cache(maxsize=1)
def get(**data):
return Settings(**data) # type: ignore[call-arg] # pydantic-setting magic
@field_validator("allowed_users")
@classmethod
def validate_allowed_users(cls, value: list[str] | str):
# Convert string to list if necessary
if isinstance(value, str):
value = value.split(",")
return value
@field_validator("tts_cache_path")
@classmethod
def validate_tts_cache_path(cls, value: Path):
# create path if it doesn't exist
value.mkdir(parents=True, exist_ok=True)
return value

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load diff

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more