Commit Graph
100 Commits
Author SHA1 Message Date
tcely 42bf4d2aac Remove fixB filter 2025-01-24 12:22:03 -05:00
tcely f109400263 Use bytesformat instead 2025-01-24 12:19:07 -05:00
tcely 840893c962 Use bytesformat instead 2025-01-24 12:15:19 -05:00
tcely 09649a3a58 Use fixB for other filesizeformat uses 2025-01-24 11:51:52 -05:00
tcely f9c1cc28d8 Use fixB filter 2025-01-24 11:44:22 -05:00
tcely 22cec87636 Add fixB filter 2025-01-24 11:37:51 -05:00
tcely 4e710f503a Create filters.py 2025-01-24 11:15:00 -05:00
tcely 1bd2e38196 Create templatetags/__init__.py 2025-01-24 10:06:24 -05:00
tcely e16f6bb86a Display database_filesize only if set 2025-01-24 09:26:04 -05:00
tcely b65ecc43ff Pass raw bytes count as database_filesize 2025-01-24 09:20:12 -05:00
tcely ee303c638b Display database_filesize 2025-01-24 09:14:53 -05:00
tcely d4a5a78831 Display format_note after audio-only formats 2025-01-24 08:26:19 -05:00
tcely 3d148fc5aa Return format_note also 2025-01-24 08:22:46 -05:00
tcely ed381715b5 Fail to previous behavior 2025-01-22 21:09:03 -05:00
tcely fb87b54300 channel_id is the only available key 2025-01-22 20:59:21 -05:00
tcely 25892388a1 Remove the /channel/ from the URL 2025-01-22 20:42:18 -05:00
tcely 5546c5dad2 Use the channel type to fetch channel_id 2025-01-22 20:21:23 -05:00
tcely e5b4e9dbc0 Add get_channel_id 2025-01-22 20:07:59 -05:00
tcely ba321945b6 Automated channel_id extraction 2025-01-22 19:41:03 -05:00
tcely 1671c6e706 DRY YouTube domain list
I am tired of links copied from YouTube not working without adjustments.
2025-01-22 04:55:57 -05:00
tcely 96078f8d40 Update architectures FAQ 2025-01-21 04:38:06 -05:00
tcely 0ea508443a Pipefile -> Pipfile 2025-01-21 04:21:47 -05:00
tcely 1f95b858f2 Use --break-system-packages with pip
Unfortunately, both versions of `pip` don't have this flag.

Check the version, then add the flag if it is not too old.
2025-01-21 03:08:24 -05:00
tcely 52d703ff1f Better check for script_name 2025-01-21 01:55:13 -05:00
tcely f65f6f1de5 Treat static_url the same as other URLs 2025-01-21 01:12:24 -05:00
tcely 66e5192980 Warn against regular updating of yt-dlp 2025-01-20 08:49:14 -05:00
tcely ae07c1ce89 Fallback to run_whl
Prefer `pip_runner` but try the `pip` wheel too, if it's missing.
2025-01-17 13:03:06 -05:00
tcely ccdd43845c Cache for pipenv & apt
- Consolidate apk & apt commands
- Consolidate ENV layers
- Consolidate RUN layers
- Remove unused variables
- Remove packages that are no longer needed
- Bind mount /app/Pipfile

`/cache` is now a `tmpfs` mount that has a `cache` mount on top for `pipenv` to use.

`apt` has `cache` mounts in the standard places:
- /var/lib/apt
- /var/cache/apt
2025-01-17 12:46:11 -05:00
tcely 4e51d54ec1 Use pip runner from pipenv 2025-01-16 08:24:11 -05:00
tcely 862c17b676 Bump ffmpeg & yt-dlp 2025-01-16 03:30:06 -05:00
tcely d7f9fa45ec Add executable to sh script 2025-01-16 08:16:31 +00:00
tcely 6f00ce8120 Create upgrade_yt-dlp.sh 2025-01-16 03:01:10 -05:00
tcely ab7b601ad2 Don't write zero into MPAA in .nfo 2025-01-16 01:31:07 -05:00
tcely 57417915bf lowercase the true value
I doubt it's unsupported by any parser, but every example uses lowercase for this, we may as well also.
2025-01-16 01:27:41 -05:00
tcely af0aae3de4 Don't write 'None' in default rating
It's better to not have a rating than to create parsing problems.

An example of what is avoided:

```
<ratings>
  <rating name="youtube" max="5" default="True">
    <value>None</value>
    <votes>16781</votes>
  </rating>
</ratings>
```
2025-01-16 01:18:20 -05:00
tcely 35f6a54823 Balance blank lines 2025-01-15 06:17:33 -05:00
tcely d349bd55c4 Avoid env for healthcheck
This is only intended for use in containers, so we know where python3 should be installed.

This is called very often, so we should try to use as few resources as we can.
2025-01-15 04:43:58 -05:00
tcely 53ad31d128 Merge branch 'main' into patch-4 2025-01-15 04:35:43 -05:00
tcely ebf9ff8eba Add environment variables for container
- TUBESYNC_SHRINK_NEW
- TUBESYNC_SHRINK_OLD

These must be set to the word 'True' (case insensitive) to enable the setting.
2025-01-15 04:26:55 -05:00
tcely 45d7039188 Only log the extra messages with the new setting 2025-01-14 05:34:59 -05:00
tcely 2860147212 Build on older docker also
* Do without --link for COPY or ADD

* Do without --checksum for ADD

* Trim the FFMPEG_VERSION variable with cut instead

I've built successfully on old Debian systems using these changes. Everything else I use has a newer docker on it.
2025-01-12 04:46:32 -05:00
tcely 5e5d011b64 Add parser directives
This hopefully helps anyone building on an older docker, such as Debian / Ubuntu packaged versions.
2025-01-12 00:49:19 -05:00
tcely ef4181c2c4 Dockerfile syntax and checks
- Specify the syntax be the latest stable version and that failed checks should stop the build.

```
By default, builds with failing build checks exit with a zero status code despite warnings. To make the build fail on warnings, set #check=error=true.
```

- Use the form of health checking that doesn't involve an extra shell on every check.
2025-01-12 00:37:30 -05:00
tcely f464acaa63 Simplify directory_path for Media 2025-01-11 15:38:45 -05:00
tcely 4364ebbff3 Multi-stage docker build for ffmpeg & s6-overlay
* Create a s6-overlay-extracted stage to copy from

This was largely inspired by: @socheatsok78

Our downloaded files are checked where that version doesn't do any verification of the downloads.

* Update ffmpeg to the first build with checksums.sha256

* Create a ffmpeg-extracted stage to copy from

* Don't preserve ownership from the builder

I was sick of the extra work with ffmpeg builds.

So, I managed to get sums generated for those builds and now we don't need to manually fill out SHA256 hashes anymore.

Now to bump ffmpeg, we can just change the date.
2025-01-11 11:53:10 -05:00
tcely 6292a9a59d Add SHRINK_OLD_MEDIA_METADATA setting 2025-01-09 14:22:37 -05:00
tcely 1ff8dfda98 Use the new copy argument to filter_response 2025-01-09 13:53:12 -05:00
tcely 274f19fa15 Use the new copy argument to filter_response 2025-01-09 13:41:23 -05:00
tcely 0f986949e5 Have filter_response return a copy, if requested 2025-01-09 13:36:43 -05:00
tcely 29c39aab1f Add SHRINK_NEW_MEDIA_METADATA setting 2025-01-09 13:20:22 -05:00
tcely 3e3f80d287 More filter_response asserts 2025-01-09 12:04:01 -05:00
tcely 4c9fa40bb0 More filter_response asserts 2025-01-09 11:47:10 -05:00
tcely 682a53da34 Add a filter_response test
First, only check that changes did happen.
2025-01-09 10:17:37 -05:00
tcely 100382f66f Rename compact_data to compact_json
This was misleading because the data dict becomes a JSON string.
2025-01-09 09:28:58 -05:00
tcely ad10bcfa61 Log both compacted and reduced sizes 2025-01-08 22:48:23 -05:00
tcely d35f52f8ac Drop /expire/ URLs from automatic_captions too 2025-01-08 11:31:23 -05:00
tcely 7b42213bbb Keep curl and add less
These are very useful when using the shell inside the container and don't use much space.
2025-01-08 00:36:37 -05:00
tcely f0b7d31949 Test and log ffmpeg version output earlier
Running the ffmpeg in an earlier (hopefully cached) layer should clean up the logs a bit.

On a related note, shadowing the environment variable was causing some confusing log output, so stop doing that as well.
2025-01-08 00:31:30 -05:00
tcely ab5e63f643 Make better use of CPU cache with nginx 2025-01-08 00:12:21 -05:00
tcely 6e116899a7 We don't need to keep bash running 2025-01-07 23:45:25 -05:00
tcely 1c432ccce1 Some formats are using a different URL 2025-01-07 13:49:58 -05:00
tcely 7793701228 Simplify results from _url_keys
Also, name the tuple values when using the results.
2025-01-07 13:01:06 -05:00
tcely 8ac5b36eee Use the exception function for traceback 2025-01-07 06:38:56 -05:00
tcely 2d85bcbe14 Use a distinct try to log errors 2025-01-07 06:20:01 -05:00
tcely c7457e94ac Check that the drop_key exists 2025-01-07 05:58:50 -05:00
tcely 7b8d11791d Drop keys from formats that cannot be useful 2025-01-07 05:39:50 -05:00
tcely 431de2e0df Loop over a set of keys for each URL type 2025-01-07 04:11:14 -05:00
tcely db25fa8029 Adjusted comment 2025-01-07 03:35:58 -05:00
tcely 9a4101a0a1 Fix formatting 2025-01-07 03:18:39 -05:00
tcely 2f34fff713 Fixes from testing
The `automatic_captions` has a layer for language codes that I didn't account for.

The type checking was copied and I didn't adjust for the arguments in this function.
2025-01-07 02:55:05 -05:00
tcely 25d2ff6802 Don't reduce the actual data yet 2025-01-07 02:12:22 -05:00
tcely ca75398548 Merge branch 'meeb:main' into filter-metadata-response 2025-01-07 01:35:13 -05:00
tcely 8c31720bf7 Log the reduction of metadata length 2025-01-07 01:33:06 -05:00
tcely 63fa97cc58 More compact JSON
The software doesn't need an extra space per key.
2025-01-07 00:43:59 -05:00
tcely 8c22b6c99e Add response filtering
These functions aren't being used yet, they will be tested against my database before that happens.
2025-01-07 00:05:57 -05:00
tcely 215aa64f2d Change to a logged warning for NFO permission problems 2025-01-06 10:35:37 -05:00
tcely e64f71a970 Don't chmod a+r when it already has those permissions 2025-01-06 10:23:17 -05:00
tcely 59865a885b Match another variation of the message
```
{key}: This video is available to this channel's members on level: Level 2: Contributor (or any higher level). Join this YouTube channel from your computer or Android app.
```
2025-01-02 02:39:19 -05:00
tcely d74e6bf2ca Avoid the unlikely possibility of None comparison 2024-12-25 20:16:07 -05:00
tcely 942452c6a2 Skip media without any formats 2024-12-25 10:53:25 -05:00
tcely 568698351d Use the N version of ffmpeg 2024-12-25 05:23:53 -05:00
tcely 3cea9270a0 Bump ffmpeg 2024-12-25 05:09:30 -05:00
tcely 8def84c7b4 Use filename_path
It's more directly what we want here.

If `filepath` ever changed to not use `filename` the value would be incorrect, so this is the cleaner way to display this value.
2024-12-25 01:17:04 -05:00
tcely 7343e39f35 Verbose output is no longer needed
I confirmed that the changes reduced the amount of files being removed.
2024-12-25 00:32:03 -05:00
tcely 0795eb951d Consistency fixes
The `filepath` variable is defined before the `if` and used throughout the first block, so we should use it in the `else` block too.

Otherwise, a change to that variable would change the first block but not the second block.
2024-12-25 00:21:58 -05:00
tcely 6c4d31765a Made messages more human friendly 2024-12-25 00:16:18 -05:00
tcely 01f0d82788 Cleanup removed media that has not been downloaded too 2024-12-25 00:07:57 -05:00
tcely 99899aac9c Use title from DB
We stored a stripped title string in the database, just use that instead of `.name` and calling `.strip()` again.
2024-12-23 20:13:56 -05:00
tcely 3b41c8df81 Remove duplicated ' to:' 2024-12-23 16:55:26 -05:00
tcely 2799d95119 Only log new media
Channels with thousands of videos, that won't be downloaded, create large blocks in the logs without this.
2024-12-23 10:17:58 -05:00
tcely b19614cc9f Do not log download progress lines by default
When you set debugging, the default progress output can be viewed and/or logged, depending on the other options.
2024-12-23 01:28:46 -05:00
tcely 67b4995448 Remove manual_skip check 2024-12-22 16:16:06 -05:00
tcely d1c6236b9d Display the location the media was actually saved to 2024-12-22 11:54:56 -05:00
tcely 8a04139da9 Add data items to make comparison easier 2024-12-22 11:46:29 -05:00
tcely 495f62f25a Display only the name part
`media.filename` is a `str`
`media.filepath` is a `Path`

The second has a simple way to remove directories, so use that instead of any custom string manipulation.
2024-12-22 11:14:32 -05:00
tcely 20527aa3c2 Update tests to not expect strings 2024-12-22 10:26:41 -05:00
tcely 8ebb333ea3 directory_path should be a Path, not str 2024-12-22 10:09:16 -05:00
tcely 4bbdfd2f24 Fix paths for when filename includes sub-directories 2024-12-22 09:44:27 -05:00
tcely a1248be4ae Update ci.yaml
Python `3.13` is not expected to work until Django `5.1` and we are currently using `3.2`, so that's not worth testing.

More info: https://code.djangoproject.com/ticket/34900
2024-12-22 05:33:08 -05:00
tcely 256f3254d9 Quote Python versions in ci.yaml 2024-12-22 05:21:28 -05:00