chore: nightly index update [skip ci]

.env.example

@@ -13,11 +13,6 @@ PINKCUFFS_USERNAME=your-email-or-username
 PINKCUFFS_PASSWORD=your-password
 PINKCUFFS_LOGIN_COOKIE=wordpress_logged_in_<hash>=<value>
 
-# femuniverse.com credentials (separate membership)
-FEMUNIVERSE_USERNAME=your-email-or-username
-FEMUNIVERSE_PASSWORD=your-password
-FEMUNIVERSE_LOGIN_COOKIE=wordpress_logged_in_<hash>=<value>
-
 # PeerTube upload target
 PEERTUBE_URL=https://your-peertube-instance.example
 PEERTUBE_USER=admin

.github/workflows/nightly-index.yml

@@ -38,8 +38,6 @@ jobs:
           JAILBIRDZ_PASSWORD: ${{ secrets.JAILBIRDZ_PASSWORD }}
           PINKCUFFS_USERNAME: ${{ secrets.PINKCUFFS_USERNAME }}
           PINKCUFFS_PASSWORD: ${{ secrets.PINKCUFFS_PASSWORD }}
-          FEMUNIVERSE_USERNAME: ${{ secrets.FEMUNIVERSE_USERNAME }}
-          FEMUNIVERSE_PASSWORD: ${{ secrets.FEMUNIVERSE_PASSWORD }}
 
       - name: Commit updated video_map.json
         if: always()  # save progress even if main.py crashed or timed out

README.md

@@ -1,6 +1,6 @@
 # 𝒥𝒶𝒾𝓁𝒷𝒾𝓇𝒹𝓏-𝒹𝓁
 
-Jailbirdz.com, Pinkcuffs.com, and Femuniverse.com are Arizona-based subscription video sites publishing arrest and jail roleplay scenarios featuring women. This tool scrapes the member area of any combination of these sites, downloads the videos, and re-hosts them on a self-owned PeerTube instance.
+Jailbirdz.com and Pinkcuffs.com are Arizona-based subscription video sites publishing arrest and jail roleplay scenarios featuring women. This tool scrapes the member area of one or both sites, downloads the videos, and re-hosts them on a self-owned PeerTube instance.
 
 > [!NOTE]  
 > This tool does not bypass authentication, modify the site, or intercept anything it isn't entitled to. A valid, paid membership is required. The scraper authenticates using your own session cookie and accesses only content your account can already view in a browser.
@@ -23,9 +23,9 @@ cp .env.example .env
 
 Set credentials for whichever sites you have a membership on. You don't need both.
 
-**Option A — credentials (recommended):** set `JAILBIRDZ_USERNAME` + `JAILBIRDZ_PASSWORD` (and/or the `PINKCUFFS_*` / `FEMUNIVERSE_*` equivalents) in `.env`. `main.py` logs in automatically on startup.
+**Option A — credentials (recommended):** set `JAILBIRDZ_USERNAME` + `JAILBIRDZ_PASSWORD` (and/or the `PINKCUFFS_*` equivalents) in `.env`. `main.py` logs in automatically on startup.
 
-**Option B — manual cookie:** set `JAILBIRDZ_LOGIN_COOKIE` (and/or `PINKCUFFS_LOGIN_COOKIE` / `FEMUNIVERSE_LOGIN_COOKIE`) yourself. Get the value from browser DevTools → Storage → Cookies — copy the full `name=value` of the `wordpress_logged_in_*` cookie.
+**Option B — manual cookie:** set `JAILBIRDZ_LOGIN_COOKIE` (and/or `PINKCUFFS_LOGIN_COOKIE`) yourself. Get the value from browser DevTools → Storage → Cookies — copy the full `name=value` of the `wordpress_logged_in_*` cookie.
 
 Sites with no credentials are skipped automatically when running `python main.py`.
 
@@ -35,8 +35,6 @@ Sites with no credentials are skipped automatically when running `python main.py
 - `JAILBIRDZ_LOGIN_COOKIE` — jailbirdz.com session cookie (fallback).
 - `PINKCUFFS_USERNAME` / `PINKCUFFS_PASSWORD` — pinkcuffs.com login.
 - `PINKCUFFS_LOGIN_COOKIE` — pinkcuffs.com session cookie (fallback).
-- `FEMUNIVERSE_USERNAME` / `FEMUNIVERSE_PASSWORD` — femuniverse.com login.
-- `FEMUNIVERSE_LOGIN_COOKIE` — femuniverse.com session cookie (fallback).
 - `PEERTUBE_URL` — base URL of your PeerTube instance.
 - `PEERTUBE_USER` — PeerTube username.
 - `PEERTUBE_CHANNEL` — channel to upload to.
@@ -52,7 +50,6 @@ Discovers all post URLs via the WordPress REST API, then visits each page with a
 python main.py                    # scrape all sites you have credentials for
 python main.py --site jailbirdz   # scrape one site only
 python main.py --site pinkcuffs --site jailbirdz  # explicit multi-site
-python main.py --site femuniverse                 # femuniverse only
 ```
 
 Results are written to `video_map.json`. Safe to re-run — already-scraped posts are skipped.
@@ -69,7 +66,7 @@ Options:
       --reorganize      Rename existing files to match current naming mode
   -w, --workers N       Concurrent downloads (default: 4)
   -n, --dry-run         Print what would be downloaded
-      --site SITE       Limit to one site (jailbirdz, pinkcuffs, or femuniverse); repeatable
+      --site SITE       Limit to one site (jailbirdz or pinkcuffs); repeatable
 ```
 
 Resumes partial downloads. The chosen naming mode is saved to `.naming_mode` inside the output directory and persists across runs. Filenames that would clash are placed into subfolders.
@@ -89,13 +86,10 @@ Options:
       --poll-interval SECS  State poll interval in seconds (default: 30)
       --skip-wait           Upload without waiting for transcoding
       --nsfw                Mark videos as NSFW
-  -n, --dry-run             Print what would be uploaded (checks the channel, uploads nothing)
-      --scan-all            Scan every subdirectory of --input, not just the site folders
+  -n, --dry-run             Print what would be uploaded
 ```
 
-Uploads in resumable 10 MB chunks. After each batch, waits for transcoding and object storage to complete before uploading the next batch — this prevents disk exhaustion on the PeerTube server. Videos already present on the channel (matched by name) are skipped. If `--input` contains any of the configured site folders (`jailbirdz`, `pinkcuffs`, `femuniverse`), only those are scanned — so the input directory can be a shared drive holding unrelated purchases alongside them. Anything skipped is listed on startup; `--scan-all` includes it. A directory with no site folders is walked in full.
-
-Progress is tracked in `.uploaded` inside the input directory, so a new `--input` directory starts with an empty ledger — the channel itself is reconciled against on every run (dry runs included) to fill it back in.
+Uploads in resumable 10 MB chunks. After each batch, waits for transcoding and object storage to complete before uploading the next batch — this prevents disk exhaustion on the PeerTube server. Videos already present on the channel (matched by name) are skipped. Progress is tracked in `.uploaded` inside the input directory.
 
 ## CI / Nightly Indexing
 
@@ -111,10 +105,6 @@ gh secret set JAILBIRDZ_PASSWORD
 # pinkcuffs (if you have a membership)
 gh secret set PINKCUFFS_USERNAME
 gh secret set PINKCUFFS_PASSWORD
-
-# femuniverse (if you have a membership)
-gh secret set FEMUNIVERSE_USERNAME
-gh secret set FEMUNIVERSE_PASSWORD
 ```
 
 **Seed CI with your current progress before the first run:**

config.py

@@ -15,9 +15,4 @@ SITES: Final[dict[str, dict[str, str]]] = {
         "cookie_domain": "pinkcuffs.com",
         "env_prefix": "PINKCUFFS",
     },
-    "femuniverse": {
-        "base_url": "https://www.femuniverse.com",
-        "cookie_domain": "femuniverse.com",
-        "env_prefix": "FEMUNIVERSE",
-    },
 }

upload.py

@@ -8,7 +8,7 @@ Usage:
     python upload.py                      # upload from ./downloads
     python upload.py -i /mnt/vol/dl      # custom input dir
     python upload.py --batch-size 2      # upload 2, wait, repeat
-    python upload.py --dry-run           # preview (checks the channel, uploads nothing)
+    python upload.py --dry-run           # preview without uploading
     python upload.py --skip-wait         # upload without waiting
 
 Required (CLI flag or env var):
@@ -32,7 +32,6 @@ import requests
 from dotenv import load_dotenv
 
 from check_clashes import fmt_size, url_to_filename, VIDEO_EXTS, load_video_map
-from config import SITES
 from download import (
     collect_urls,
     get_paths_for_mode,
@@ -50,7 +49,6 @@ DEFAULT_BATCH_SIZE = 1
 DEFAULT_POLL = 30
 UPLOADED_FILE = ".uploaded"
 PT_NAME_MAX = 120
-PT_DESC_MIN = 3  # PeerTube rejects descriptions shorter than this
 
 
 # ── Text helpers ─────────────────────────────────────────────────────
@@ -64,11 +62,6 @@ def clean_description(raw: str) -> str:
     text = re.sub(r"<[^>]+>", "", text)
     text = html.unescape(text)
     text = re.sub(r"\n{3,}", "\n\n", text).strip()
-    # PeerTube enforces a 3-char minimum on descriptions; a sub-minimum
-    # description (e.g. a stray ".") makes the upload-init 400.  Drop it so
-    # it's omitted from the request rather than rejected.
-    if len(text) < PT_DESC_MIN:
-        return ""
     return text[:10000]
 
 
@@ -449,43 +442,14 @@ def build_path_to_meta(
     return result
 
 
-def site_dirs_in(input_dir: str) -> set[str]:
-    """Names of the per-site subdirectories that actually exist under input_dir."""
-    root = Path(input_dir)
-    if not root.is_dir():
-        return set()
-    return {site for site in SITES if (root / site).is_dir()}
-
-
-def unmanaged_dirs_in(input_dir: str, site_dirs: set[str]) -> list[str]:
-    """Top-level subdirectories of input_dir that belong to no configured site."""
-    root = Path(input_dir)
-    if not root.is_dir():
-        return []
-    return sorted(
-        d.name
-        for d in root.iterdir()
-        if d.is_dir() and not d.name.startswith(".") and d.name not in site_dirs
-    )
-
-
-def find_videos(input_dir: str, only_dirs: set[str] | None = None) -> set[Path]:
-    """Walk input_dir and return a set of relative paths for all video files.
-
-    When *only_dirs* is given, just those top-level directories are descended
-    into; loose files sitting directly in input_dir are always included.  That
-    keeps a shared parent directory — an external drive holding other purchases
-    next to the site folders — from dragging in unrelated albums.
-    """
-    top = Path(input_dir)
+def find_videos(input_dir: str) -> set[Path]:
+    """Walk input_dir and return a set of relative paths for all video files."""
     found = set()
-    for cur, dirs, files in os.walk(str(top)):
+    for root, dirs, files in os.walk(input_dir):
         dirs[:] = [d for d in dirs if not d.startswith(".")]
-        if only_dirs is not None and Path(cur) == top:
-            dirs[:] = [d for d in dirs if d in only_dirs]
         for f in files:
             if Path(f).suffix.lower() in VIDEO_EXTS:
-                found.add((Path(cur) / f).relative_to(top))
+                found.add((Path(root) / f).relative_to(input_dir))
     return found
 
 
@@ -562,11 +526,6 @@ def main() -> None:
     ap.add_argument(
         "--dry-run", "-n", action="store_true", help="Preview what would be uploaded"
     )
-    ap.add_argument(
-        "--scan-all",
-        action="store_true",
-        help="Scan every subdirectory of --input, not just the configured site folders",
-    )
     args = ap.parse_args()
 
     url = args.url or os.environ.get("PEERTUBE_URL")
@@ -574,42 +533,26 @@ def main() -> None:
     channel = args.channel or os.environ.get("PEERTUBE_CHANNEL")
     password = args.password or os.environ.get("PEERTUBE_PASSWORD")
 
-    missing = [
-        label
-        for label, val in [
-            ("--url / PEERTUBE_URL", url),
-            ("--username / PEERTUBE_USER", username),
-            ("--channel / PEERTUBE_CHANNEL", channel),
-            ("--password / PEERTUBE_PASSWORD", password),
+    if not args.dry_run:
+        missing = [
+            label
+            for label, val in [
+                ("--url / PEERTUBE_URL", url),
+                ("--username / PEERTUBE_USER", username),
+                ("--channel / PEERTUBE_CHANNEL", channel),
+                ("--password / PEERTUBE_PASSWORD", password),
+            ]
+            if not val
         ]
-        if not val
-    ]
-    have_creds = not missing
-    if missing and not args.dry_run:
-        for label in missing:
-            print(f"[!] Required: {label}")
-        sys.exit(1)
+        if missing:
+            for label in missing:
+                print(f"[!] Required: {label}")
+            sys.exit(1)
 
     # ── load metadata & scan disk ──
     video_map = load_video_map()
     path_meta = build_path_to_meta(video_map, args.input)
-
-    # --input may be a shared directory that holds unrelated purchases beside
-    # the per-site folders.  Once any site folder is present, treat the layout
-    # as site-partitioned and ignore everything else; a flat directory (no site
-    # folders at all) is still walked in full.
-    site_dirs = set() if args.scan_all else site_dirs_in(args.input)
-    if site_dirs:
-        skipped = unmanaged_dirs_in(args.input, site_dirs)
-        if skipped:
-            print(
-                f"[!] Ignoring {len(skipped)} unmanaged folder(s) in {args.input}: "
-                f"{', '.join(skipped)}"
-            )
-            print("    (pass --scan-all to upload those too)")
-        on_disk = find_videos(args.input, site_dirs)
-    else:
-        on_disk = find_videos(args.input)
+    on_disk = find_videos(args.input)
 
     unmatched = on_disk - set(path_meta.keys())
     if unmatched:
@@ -619,12 +562,11 @@ def main() -> None:
         for rel in unmatched:
             path_meta[rel] = {"title": "", "description": ""}
 
-    ledger = Path(args.input) / UPLOADED_FILE
     uploaded = load_uploaded(args.input)
     pending = sorted(rel for rel in on_disk if rel not in uploaded)
 
     print(f"[+] {len(on_disk)} video files in {args.input}/")
-    print(f"[+] {len(uploaded)} already uploaded (per {ledger})")
+    print(f"[+] {len(uploaded)} already uploaded")
     print(f"[+] {len(pending)} pending")
     print(f"[+] Batch size: {args.batch_size}")
 
@@ -632,79 +574,6 @@ def main() -> None:
         print("\nAll videos already uploaded.")
         return
 
-    # ── connect to the channel ────────────────────────────────────────
-    # The .uploaded ledger lives inside --input, so pointing at a fresh
-    # directory (an external drive, say) starts from an empty one.  The channel
-    # is the real source of truth, so reconcile against it before deciding what
-    # is pending — dry runs included, whenever credentials are available.
-    base = ""
-    token = ""
-    channel_id = 0
-    existing: set[str] = set()
-
-    if have_creds:
-        assert url is not None
-        assert username is not None
-        assert channel is not None
-        assert password is not None
-
-        base = url.rstrip("/")
-        if not base.startswith("http"):
-            base = "https://" + base
-
-        print(f"\n[+] Authenticating with {base} ...")
-        token = get_oauth_token(base, username, password)
-        print(f"[+] Authenticated as {username}")
-
-        channel_id = get_channel_id(base, token, channel)
-        print(f"[+] Channel: {channel} (id {channel_id})")
-
-        name_counts = get_channel_video_names(base, token, channel)
-        existing = set(name_counts)
-        total = sum(name_counts.values())
-        print(
-            f"[+] Found {total} video(s) on channel ({len(name_counts)} unique name(s))"
-        )
-
-        dupes = {name: count for name, count in name_counts.items() if count > 1}
-        if dupes:
-            print(f"[!] {len(dupes)} duplicate name(s) detected on channel:")
-            for name, count in sorted(dupes.items()):
-                print(f"    x{count}  {name}")
-    else:
-        print(
-            "\n[!] No PeerTube credentials given — skipping the channel check; "
-            "everything below is reported as pending."
-        )
-
-    # ── pre-reconcile: sweep all pending against channel names ────────
-    # The main upload loop discovers already-uploaded videos lazily as it
-    # walks the sorted pending list — meaning on a fresh run (no .uploaded
-    # file) you won't know how many files are genuinely new until the loop
-    # has processed everything.  Doing a full sweep here, before any
-    # upload starts, gives an accurate count up-front and pre-populates
-    # .uploaded so that interrupted/re-run sessions skip them instantly
-    # without re-checking each time.  A dry run sweeps but writes nothing.
-    if existing:
-        pre_matched = [
-            rel for rel in pending if _channel_match(rel, path_meta, existing)[0]
-        ]
-        if pre_matched:
-            suffix = "" if args.dry_run else " — marking uploaded"
-            print(
-                f"\n[+] Pre-sweep: {len(pre_matched)} local file(s) already on channel{suffix}"
-            )
-            if not args.dry_run:
-                for rel in pre_matched:
-                    mark_uploaded(args.input, rel)
-            matched = set(pre_matched)
-            pending = [rel for rel in pending if rel not in matched]
-            print(f"[+] {len(pending)} left to upload\n")
-
-    if not pending:
-        print("All videos already uploaded.")
-        return
-
     # ── dry run ──
     if args.dry_run:
         total_bytes = 0
@@ -717,6 +586,55 @@ def main() -> None:
         print(f"\n  Total: {fmt_size(total_bytes)} across {len(pending)} videos")
         return
 
+    assert url is not None
+    assert username is not None
+    assert channel is not None
+    assert password is not None
+
+    # ── authenticate ──
+    base = url.rstrip("/")
+    if not base.startswith("http"):
+        base = "https://" + base
+
+    print(f"\n[+] Authenticating with {base} ...")
+    token = get_oauth_token(base, username, password)
+    print(f"[+] Authenticated as {username}")
+
+    channel_id = get_channel_id(base, token, channel)
+    print(f"[+] Channel: {channel} (id {channel_id})")
+
+    name_counts = get_channel_video_names(base, token, channel)
+    existing = set(name_counts)
+    total = sum(name_counts.values())
+    print(f"[+] Found {total} video(s) on channel ({len(name_counts)} unique name(s))")
+
+    dupes = {name: count for name, count in name_counts.items() if count > 1}
+    if dupes:
+        print(f"[!] {len(dupes)} duplicate name(s) detected on channel:")
+        for name, count in sorted(dupes.items()):
+            print(f"    x{count}  {name}")
+
+    # ── pre-reconcile: sweep all pending against channel names ────────
+    # The main upload loop discovers already-uploaded videos lazily as it
+    # walks the sorted pending list — meaning on a fresh run (no .uploaded
+    # file) you won't know how many files are genuinely new until the loop
+    # has processed everything.  Doing a full sweep here, before any
+    # upload starts, gives an accurate count up-front and pre-populates
+    # .uploaded so that interrupted/re-run sessions skip them instantly
+    # without re-checking each time.
+    pre_matched = []
+    for rel in pending:
+        if _channel_match(rel, path_meta, existing)[0]:
+            pre_matched.append(rel)
+    if pre_matched:
+        print(
+            f"\n[+] Pre-sweep: {len(pre_matched)} local file(s) already on channel — marking uploaded"
+        )
+        for rel in pre_matched:
+            mark_uploaded(args.input, rel)
+        pending = [rel for rel in pending if rel not in set(pre_matched)]
+        print(f"[+] {len(pending)} left to upload\n")
+
     nsfw = args.nsfw
     total_up = 0
     batch: list[tuple[str, str]] = []  # [(uuid, name), ...]
@@ -779,10 +697,4 @@ def main() -> None:
 
 
 if __name__ == "__main__":
-    try:
-        main()
-    except BrokenPipeError:
-        # Downstream reader closed the pipe (e.g. `| head`).  Point stdout at
-        # devnull so interpreter shutdown does not raise a second time.
-        os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
-        sys.exit(0)
+    main()