wimby/tubesync
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
929 Commits 2 Branches 0 Tags
152f88e7e70aabad64d824212c402d07e18838d1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
tcely b816eb3596 fixup: use the name attribute 2025-01-29 04:05:18 -05:00
tcely 78b9aadcbf Use a temporary directory per task 
I am seeing multiple ffmpeg processes that mess each other up.

I do not know why this is happening yet, but for now let them operate in their own temporary directories.
 2025-01-29 03:55:52 -05:00
tcely 3241f5dde6 Merge branch 'main' into issue-645-sponsorblock 2025-01-29 01:05:01 -05:00
tcely 24ac2e1aed Field expects verbose_name to be first, not separator 2025-01-29 00:23:02 -05:00
tcely f60c1d7b5b Move SponsorBlock_Category to fields.py 2025-01-28 21:53:42 -05:00
tcely 2b5ad21b20 Use TextChoices for SponsorBlock_Category 2025-01-28 20:37:32 -05:00
tcely e3501fe77a Fix CommaSepChoiceField 
Using the separator that was chosen is the biggest fix.
 2025-01-28 19:12:56 -05:00
tcely e59ba88e4e Add default values for the new settings 2025-01-28 16:29:45 -05:00
tcely d9c73d3dd7 Add new container settings 
- TUBESYNC_RENAME_ALL_SOURCES: True or False
- TUBESYNC_RENAME_SOURCES: A comma-separated list of Source directories
 2025-01-28 16:28:21 -05:00
tcely bf7127b973 Schedule the rename task when only the settings allow it 2025-01-28 16:19:17 -05:00
tcely 0695036610 Tweak the behavior of post processors 2025-01-28 01:45:45 -05:00
tcely 803319fe9f Let deepcopy do more work for us 2025-01-28 01:00:35 -05:00
tcely b40626f9dc Guess at ffmpeg options based on filename 2025-01-27 20:24:23 -05:00
tcely d34e1a204e Set a default nice level 
Now that SponsorBlock is running, these jobs are using significant CPU resources.

Setting a nice level to let other things run better while `ffmpeg` is running seems wise.
 2025-01-27 16:12:52 -05:00
tcely 133f08bd86 Hide format_note for combined audio / video 2025-01-27 15:17:39 -05:00
tcely bb226d4bc1 Made the output more friendly 
- Humanize the Hz number 
- Do not output '@Nonek' for combined audio / video
 2025-01-27 15:08:24 -05:00
tcely d2641c39cd Add asr to parsed formats 2025-01-27 14:34:10 -05:00
tcely ec16e4c54d Use the parsed formats 2025-01-27 14:31:16 -05:00
tcely eb27bf1583 Only save() the current Media 2025-01-27 13:17:16 -05:00
tcely d160a043e5 Use yt_dlp.get_postprocessors() 2025-01-27 12:45:32 -05:00
tcely 8a3a7aa333 Display language_code for audio formats 2025-01-27 03:50:42 -05:00
tcely 9fc773b5c0 Add language_code from response 2025-01-27 03:43:56 -05:00
tcely 09ecbf210e Default to valid JSON 
This prevents useless logged errors when there is no metadata available yet.
 2025-01-27 02:57:47 -05:00
tcely a44e856378 Merge branch 'main' into rename-files-with-source-format-issue-185 2025-01-26 10:17:12 -05:00
tcely bcbafaf842 Remove directory manipulation 
Added a comment about why these directories aren't matching.
 2025-01-26 07:54:14 -05:00
tcely 7b1aedf7ae Put os.path.basename() back 
Without this the subdirectory from `media_format` is doubled.
 2025-01-26 07:11:30 -05:00
tcely fe4f6dd6b8 DRY {JSON,nfo,thumb}name functions 2025-01-25 22:26:39 -05:00
tcely 31e25eb22b Remove unnecessary spaces 2025-01-25 20:21:59 -05:00
tcely 7169ba5865 Remove unnecessary spaces 2025-01-25 20:14:46 -05:00
tcely 32fad91401 Remove unnecessary spaces 2025-01-25 20:12:55 -05:00
tcely de1490c416 Remove unnecessary spaces 2025-01-25 20:11:19 -05:00
tcely b319101c70 Remove unnecessary spaces 2025-01-25 20:08:10 -05:00
tcely 69e895818e Drop the thumbnail task if media downloaded first 2025-01-25 19:46:58 -05:00
tcely fe39717985 Change media download priority 2025-01-25 19:23:52 -05:00
tcely 3b20f07450 Try to download thumbnail before copy 2025-01-25 19:12:36 -05:00
tcely 56e1aae5cd Simplify get_queryset 2025-01-25 09:46:03 -05:00
tcely e16f6bb86a Display database_filesize only if set 2025-01-24 09:26:04 -05:00
tcely b65ecc43ff Pass raw bytes count as database_filesize 2025-01-24 09:20:12 -05:00
tcely ee303c638b Display database_filesize 2025-01-24 09:14:53 -05:00
tcely d4a5a78831 Display format_note after audio-only formats 2025-01-24 08:26:19 -05:00
tcely 3d148fc5aa Return format_note also 2025-01-24 08:22:46 -05:00
tcely ed381715b5 Fail to previous behavior 2025-01-22 21:09:03 -05:00
tcely fb87b54300 channel_id is the only available key 2025-01-22 20:59:21 -05:00
tcely 25892388a1 Remove the /channel/ from the URL 2025-01-22 20:42:18 -05:00
tcely 5546c5dad2 Use the channel type to fetch channel_id 2025-01-22 20:21:23 -05:00
tcely e5b4e9dbc0 Add get_channel_id 2025-01-22 20:07:59 -05:00
tcely ba321945b6 Automated channel_id extraction 2025-01-22 19:41:03 -05:00
tcely 1671c6e706 DRY YouTube domain list 
I am tired of links copied from YouTube not working without adjustments.
 2025-01-22 04:55:57 -05:00
tcely 96078f8d40 Update architectures FAQ 2025-01-21 04:38:06 -05:00
tcely 0ea508443a Pipefile -> Pipfile 2025-01-21 04:21:47 -05:00
tcely 1f95b858f2 Use --break-system-packages with pip 
Unfortunately, both versions of `pip` don't have this flag.

Check the version, then add the flag if it is not too old.
 2025-01-21 03:08:24 -05:00
tcely 52d703ff1f Better check for script_name 2025-01-21 01:55:13 -05:00
tcely f65f6f1de5 Treat static_url the same as other URLs 2025-01-21 01:12:24 -05:00
tcely 66e5192980 Warn against regular updating of yt-dlp 2025-01-20 08:49:14 -05:00
tcely ae07c1ce89 Fallback to run_whl 
Prefer `pip_runner` but try the `pip` wheel too, if it's missing.
 2025-01-17 13:03:06 -05:00
tcely ccdd43845c Cache for pipenv & apt 
- Consolidate apk & apt commands
- Consolidate ENV layers
- Consolidate RUN layers
- Remove unused variables
- Remove packages that are no longer needed
- Bind mount /app/Pipfile

`/cache` is now a `tmpfs` mount that has a `cache` mount on top for `pipenv` to use.

`apt` has `cache` mounts in the standard places:
- /var/lib/apt
- /var/cache/apt
 2025-01-17 12:46:11 -05:00
tcely 4e51d54ec1 Use pip runner from pipenv 2025-01-16 08:24:11 -05:00
tcely 862c17b676 Bump ffmpeg & yt-dlp 2025-01-16 03:30:06 -05:00
tcely d7f9fa45ec Add executable to sh script 2025-01-16 08:16:31 +00:00
tcely 6f00ce8120 Create upgrade_yt-dlp.sh 2025-01-16 03:01:10 -05:00
tcely ab7b601ad2 Don't write zero into MPAA in .nfo 2025-01-16 01:31:07 -05:00
tcely 57417915bf lowercase the true value 
I doubt it's unsupported by any parser, but every example uses lowercase for this, we may as well also.
 2025-01-16 01:27:41 -05:00
tcely af0aae3de4 Don't write 'None' in default rating 
It's better to not have a rating than to create parsing problems.

An example of what is avoided:

```
<ratings>
  <rating name="youtube" max="5" default="True">
    <value>None</value>
    <votes>16781</votes>
  </rating>
</ratings>
```
 2025-01-16 01:18:20 -05:00
tcely 35f6a54823 Balance blank lines 2025-01-15 06:17:33 -05:00
tcely d349bd55c4 Avoid env for healthcheck 
This is only intended for use in containers, so we know where python3 should be installed.

This is called very often, so we should try to use as few resources as we can.
 2025-01-15 04:43:58 -05:00
tcely 53ad31d128 Merge branch 'main' into patch-4 2025-01-15 04:35:43 -05:00
tcely ebf9ff8eba Add environment variables for container 
- TUBESYNC_SHRINK_NEW
- TUBESYNC_SHRINK_OLD

These must be set to the word 'True' (case insensitive) to enable the setting.
 2025-01-15 04:26:55 -05:00
tcely 45d7039188 Only log the extra messages with the new setting 2025-01-14 05:34:59 -05:00
tcely 2860147212 Build on older docker also 
* Do without --link for COPY or ADD

* Do without --checksum for ADD

* Trim the FFMPEG_VERSION variable with cut instead

I've built successfully on old Debian systems using these changes. Everything else I use has a newer docker on it.
 2025-01-12 04:46:32 -05:00
tcely 5e5d011b64 Add parser directives 
This hopefully helps anyone building on an older docker, such as Debian / Ubuntu packaged versions.
 2025-01-12 00:49:19 -05:00
tcely ef4181c2c4 Dockerfile syntax and checks 
- Specify the syntax be the latest stable version and that failed checks should stop the build.

```
By default, builds with failing build checks exit with a zero status code despite warnings. To make the build fail on warnings, set #check=error=true.
```

- Use the form of health checking that doesn't involve an extra shell on every check.
 2025-01-12 00:37:30 -05:00
tcely f464acaa63 Simplify directory_path for Media 2025-01-11 15:38:45 -05:00
tcely 4364ebbff3 Multi-stage docker build for ffmpeg & s6-overlay 
* Create a s6-overlay-extracted stage to copy from

This was largely inspired by: @socheatsok78

Our downloaded files are checked where that version doesn't do any verification of the downloads.

* Update ffmpeg to the first build with checksums.sha256

* Create a ffmpeg-extracted stage to copy from

* Don't preserve ownership from the builder

I was sick of the extra work with ffmpeg builds.

So, I managed to get sums generated for those builds and now we don't need to manually fill out SHA256 hashes anymore.

Now to bump ffmpeg, we can just change the date.
 2025-01-11 11:53:10 -05:00
tcely 6292a9a59d Add SHRINK_OLD_MEDIA_METADATA setting 2025-01-09 14:22:37 -05:00
tcely 1ff8dfda98 Use the new copy argument to filter_response 2025-01-09 13:53:12 -05:00
tcely 274f19fa15 Use the new copy argument to filter_response 2025-01-09 13:41:23 -05:00
tcely 0f986949e5 Have filter_response return a copy, if requested 2025-01-09 13:36:43 -05:00
tcely 29c39aab1f Add SHRINK_NEW_MEDIA_METADATA setting 2025-01-09 13:20:22 -05:00
tcely 3e3f80d287 More filter_response asserts 2025-01-09 12:04:01 -05:00
tcely 4c9fa40bb0 More filter_response asserts 2025-01-09 11:47:10 -05:00
tcely 682a53da34 Add a filter_response test 
First, only check that changes did happen.
 2025-01-09 10:17:37 -05:00
tcely 100382f66f Rename compact_data to compact_json 
This was misleading because the data dict becomes a JSON string.
 2025-01-09 09:28:58 -05:00
tcely ad10bcfa61 Log both compacted and reduced sizes 2025-01-08 22:48:23 -05:00
tcely d35f52f8ac Drop /expire/ URLs from automatic_captions too 2025-01-08 11:31:23 -05:00
tcely 7b42213bbb Keep curl and add less 
These are very useful when using the shell inside the container and don't use much space.
 2025-01-08 00:36:37 -05:00
tcely f0b7d31949 Test and log ffmpeg version output earlier 
Running the ffmpeg in an earlier (hopefully cached) layer should clean up the logs a bit.

On a related note, shadowing the environment variable was causing some confusing log output, so stop doing that as well.
 2025-01-08 00:31:30 -05:00
tcely ab5e63f643 Make better use of CPU cache with nginx 2025-01-08 00:12:21 -05:00
tcely 6e116899a7 We don't need to keep bash running 2025-01-07 23:45:25 -05:00
tcely 1c432ccce1 Some formats are using a different URL 2025-01-07 13:49:58 -05:00
tcely 7793701228 Simplify results from _url_keys 
Also, name the tuple values when using the results.
 2025-01-07 13:01:06 -05:00
tcely 8ac5b36eee Use the exception function for traceback 2025-01-07 06:38:56 -05:00
tcely 2d85bcbe14 Use a distinct try to log errors 2025-01-07 06:20:01 -05:00
tcely c7457e94ac Check that the drop_key exists 2025-01-07 05:58:50 -05:00
tcely 7b8d11791d Drop keys from formats that cannot be useful 2025-01-07 05:39:50 -05:00
tcely 431de2e0df Loop over a set of keys for each URL type 2025-01-07 04:11:14 -05:00
tcely db25fa8029 Adjusted comment 2025-01-07 03:35:58 -05:00
tcely 9a4101a0a1 Fix formatting 2025-01-07 03:18:39 -05:00
tcely 2f34fff713 Fixes from testing 
The `automatic_captions` has a layer for language codes that I didn't account for.

The type checking was copied and I didn't adjust for the arguments in this function.
 2025-01-07 02:55:05 -05:00
tcely 25d2ff6802 Don't reduce the actual data yet 2025-01-07 02:12:22 -05:00
tcely ca75398548 Merge branch 'meeb:main' into filter-metadata-response 2025-01-07 01:35:13 -05:00