Tits In Tops Thread Scraper Tool / How to download TiTs pictures?

cocopopper000 said:
Is there a way to automatically download all of the full size picks from the posts here so they can be viewed all at once without having to keep opening and closing things or saving them one by one? I normally use Bulk Image Downloader or JDownloader2, but they seem not to work with this forum.
Click to expand...

DownThemAll! works. Use the 'Links' tab, as the 'Media' tab will just download the thumbnails. Putting att in the filter and checking 'Disable others' tells it to only download images; and putting data tells it to only download videos. Note that you need to have 'All files' selected, not images or videos, because the app does not recognize our attachments as images because their URLs don't end in a file extension. So, following these instructions will enable you to download all attachments in a thread. If you only want to download a certain file type, you can put the file type instead of attachments, e.g. jpg or mp4. I've also discovered that you can use DownThemAll! to download an entire thread: it requires you to open all pages of a thread in separate tabs and click the DownThemAll! - All Tabs option. Obviously that should be done in window that contains only the tabs that you want to download from.

I just tested it in this thread and it indeed downloads all 4 of the attachments, the only problem is it also downloads a useless HTML file because it contains the word attachments.* But obviously deleting a single file after the download is nothing to worry about.


*This is the link to the file, in case anyone is curious:

Code: 
https://titsintops.com/phpBB2/index.php?attachments/upload&type=post&context[post_id]=1463815&hash=92152f6fe0c3f79c7bd73f34b15605f9
 
Last edited:
Update on the thread scraper: fix for new html format (IE updated CSS selector for file attachments).
 

Attachments

  • tit-thread-scraper.zip
    5.3 KB · Views: 161
somber said:
I'm getting "Request failed with code 404", but I swear my link is valid:
https://www.titsintops.com/phpBB2/i...nique-aka-minijello-mini-jello-topless.96304/

Running the build from Aug 1. Am I stupidly missing something?

python3.9 main.py -u "myEmail@gmail.com" -p "myPass" "https://titsintops.com/phpBB2/index...nique-aka-minijello-mini-jello-topless.96304/"

Python 3.9.5 (default, May 19 2021, 11:32:47)
[GCC 9.3.0]
Click to expand...
This isn't working for me either. They might have changed something on the server side, so I'll have to debug it. Hopefully I can get around to it tonight. Also thanks for reminding me to scrape minijello's thread, can't believe I haven't yet. I'll let you know when I've figured it out.
 
mason2371 said:
This isn't working for me either. They might have changed something on the server side, so I'll have to debug it. Hopefully I can get around to it tonight. Also thanks for reminding me to scrape minijello's thread, can't believe I haven't yet. I'll let you know when I've figured it out.
Click to expand...
You're sincerely a legend, thank you!
 
mason2371 said:
This isn't working for me either. They might have changed something on the server side, so I'll have to debug it. Hopefully I can get around to it tonight. Also thanks for reminding me to scrape minijello's thread, can't believe I haven't yet. I'll let you know when I've figured it out.
Click to expand...
That wasn't so bad. Two issues fixed:
  1. URL normalizing was just basic string operations, which screwed up on https://www.titsintops.com links (and not just plain https://titsintops.com). Replaced with actual url parsing, so either form should work.
    1. Side note: if the web admin ever decides to properly configure the bare domain to www redirect, I'll have to change this, but since all my archive files are using the bare domain I'm keeping it that way.
  2. Some media files (confirmed on an embedded video on page 27 of the minijello thread) were breaking with a trailing slash at the end of the url, despite every other url on this site working. Added a .strip("/") to the media url pulling function.
tl;dr fixed the shit that was broken

PS let me know if anything isn't working. Just pulled the minijello thread and everything seemed fine.

Edit: forgot to tag @somber
 

Attachments

  • tit-thread-scraper.zip
    5.3 KB · Views: 212
mason2371 said:
That wasn't so bad. Two issues fixed:
  1. URL normalizing was just basic string operations, which screwed up on https://www.titsintops.com links (and not just plain https://titsintops.com). Replaced with actual url parsing, so either form should work.
    1. Side note: if the web admin ever decides to properly configure the bare domain to www redirect, I'll have to change this, but since all my archive files are using the bare domain I'm keeping it that way.
  2. Some media files (confirmed on an embedded video on page 27 of the minijello thread) were breaking with a trailing slash at the end of the url, despite every other url on this site working. Added a .strip("/") to the media url pulling function.
tl;dr fixed the shit that was broken

PS let me know if anything isn't working. Just pulled the minijello thread and everything seemed fine.

Edit: forgot to tag @somber
Click to expand...
Well, shit. I didn't realize that the new url normalization would add a slash to the end of media urls, which makes them not match some entries in archive files. Word of warning while I go clean out a couple gigs of duplicate files. Everything still works, but you'll have to either re-scrape threads from scratch or remove any duplicates after scraping with the new version, or make sure every line in archive files you have ends in '/'.

If you're on linux, you probably don't need me to tell you this, but:
Bash: 
find /path/to/porn/folder -type f -name archive.tit.txt -print0 | xargs -0 sed -si '/[^/]$/ { s/$/\//g }'
If you're on Windows, good luck with your GUIs, or just re-scrape everything (sorry admins).

Edit: fixed sed script, last version replaced the last character with slashes instead of appending
 
Last edited:
@mason2371 Fucking amazing, thank you!

I haven't scraped anything yet, so should be all good regarding the archive! :)

Edit:
I'm still getting a 404. Trying to figure out where I'm going wrong, as it's likely my side now since you mentioned you didn't have any issues. I realized that I included www. when posting out the URL in my comment, but I wasn't including it in the actual command (and didn't in the command in my comment). I've tried both, still no luck :'(

2nd Edit:

Checked the bad URL in the output - I somehow picked up a rogue backslash: https://titsintops.com/phpBB2/index.php\?threads/busty-dominique-aka-minijello-mini-jello-topless.96304/

Removed it, now I'm good. Thank's again
 
Last edited:
infinitedrive06 said:
Any way to get a small tutorial for us normies? I can follow instructions but don't know where to go with this right now
Click to expand...
Installing:
  1. Make sure you have Python 3 installed. If not, go get it. Make sure to check the installer box for pip and add to PATH on Windows, Mac don't know what the installer is like so you're on your own
  2. Download and unpack the script zip file to wherever you want
  3. Open a terminal (cmd for windows, Terminal for Mac)
  4. In the terminal, 'cd' to wherever you unpacked the script to. E.g: cd "C:\Users\YOU\Downloads\tit-thread-scraper"
  5. Run: 'pip install -r requirements.txt'
    1. If you don't have permissions, try opening cmd as an admin (windows) or using 'sudo pip' instead of just 'pip' (mac)
Using:
  1. Open your terminal again (or use the same window from installation) and this time cd to the folder you want to scrape everything into. Personally, I do one folder per thread/girl.
  2. Run python /path/to/scraper/main.py with the options you want.
Notes:
  • You can rename the main.py script to whatever you want, and once you run pip install you can delete requirements.txt
  • You can put the script somewhere safe and add its location to your PATH so you can run the script from anywhere without putting in the full path to it. Google "add folder to path <your os>". For this you'll also have to make the script executable. On Mac, use chmod and you may have to edit the #! line at the top of the script to point to your python. On Windows, I believe you just have to set the default program for opening .py files to python. Or, just type out the whole damn path every time you run the script.
  • Run main.py --help to see what the options are
  • From the help screen, options can be short (-h) or long (--help), pick whichever you prefer
  • The clobber option will prevent duplicate files, but if two different images were uploaded with the same name to a thread the second one will overwrite the first. I never use it myself
  • The links option only works when people take the time and use the insert link function in their posts, so it's not perfect
  • The archive option is used to prevent re-downloading files when you run the script again. Just set the same file for both runs
  • The output directory defaults to the folder you are in in your terminal. If you enter just 'cd' into Windows cmd it'll tell you where that is. Mac is probably 'pwd'. Use 'cd <path>' to change that on Windows or Mac. When you specify a folder for --output-directory, it's relative to your current folder in your terminal, unless you start it with 'C:\' (windows) or '/' (mac)
Examples:
To scrape Merilyn Sakova on Windows:
Bash: 
cd "C:\Users\YOU\Documents\Secret Porn Folder\Merilyn Sakova"
python C:\Users\YOU\Desktop\main.py --username "you@username.com" --password "yourTiTPassword" --archive archive.txt --links urls.unsorted.txt https://titsintops.com/phpBB2/index.php?threads/busty-anya-merilyn-sakova-appreciation-thread.617

The script I use on linux (this won't work the same on Windows, but it'll give you ideas if you want to hack on it yourself):
Bash: 
URL_FILENAME='tit.url'

thread="$1"
if [ -z "$thread" ]; then
        if [ -s "$URL_FILENAME" ]; then
                thread=$(cat "$URL_FILENAME")
        else
                echo 'Error: no thread specified, and url file not found, or file is empty' 1>&2
                exit 1
        fi
fi
username='FIXME'
password='FIXME'
touch . && \
mkdir -p tit && \
~/Documents/dev/tit-thread-scraper/src/main.py \
        --username "$username" \
        --password "$password" \
        --links urls.unsorted \
        --archive archive.tit.txt \
        --output-directory tit \
        "$thread"

Note on the quotes in this post:
'Single quotes' indicate a term, but don't actually include them when you enter into your terminal
"Double quotes" should be included when you run commands, since spaces are interpreted by the terminal as option separators if not in quotes

Let me know if you need more help
 
JTB57 said:
DownThemAll! works. Use the 'Links' tab, as the 'Media' tab will just download the thumbnails. Putting attachments in the filter and checking 'Disable others' tells it to only download attachments. Edit: You need to have 'All files' selected, not images or videos, because the app does not recognize our attachments as images because their URLs don't end in a file extension. So, following these instructions will enable you to download all attachments in a thread. If you only want to download a certain file type, you can put the file type instead of attachments, e.g. jpg or mp4. I've also discovered that you can use DownThemAll! to download an entire thread: it requires you to open all pages of a thread in separate tabs and click the DownThemAll! - All Tabs option. Obviously that should be done in window that contains only the tabs that you want to download from.

I just tested it in this thread and it indeed downloads all 4 of the attachments, the only problem is it also downloads a useless HTML file because it contains the word attachments.* But obviously deleting a single file after the download is nothing to worry about.


*This is the link to the file, in case anyone is curious:

Code: 
https://titsintops.com/phpBB2/index.php?attachments/upload&type=post&context[post_id]=1463815&hash=92152f6fe0c3f79c7bd73f34b15605f9
Click to expand...
Thanks for this tip. Unfortunately it does not seem to work anymore, it says that the access is restricted for the "attachments" and it will not download them.
 
Idea: Make a version of this that saves the entire thread (posts and uploads). The text on this forum is important too.

On a desktop, a person can save each page to his computer, but none of the images/videos will be clickable...

chappelle said:
Thanks for this tip. Unfortunately it does not seem to work anymore, it says that the access is restricted for the "attachments" and it will not download them.
Click to expand...

It still works. I've used it several times in the past few days. I have a feeling that your problem was that you weren't logged in!
 
JTB57 said:
DownThemAll! works. Use the 'Links' tab, as the 'Media' tab will just download the thumbnails. Putting attachments in the filter and checking 'Disable others' tells it to only download attachments. Edit: You need to have 'All files' selected, not images or videos, because the app does not recognize our attachments as images because their URLs don't end in a file extension. So, following these instructions will enable you to download all attachments in a thread. If you only want to download a certain file type, you can put the file type instead of attachments, e.g. jpg or mp4. I've also discovered that you can use DownThemAll! to download an entire thread: it requires you to open all pages of a thread in separate tabs and click the DownThemAll! - All Tabs option. Obviously that should be done in window that contains only the tabs that you want to download from.

I just tested it in this thread and it indeed downloads all 4 of the attachments, the only problem is it also downloads a useless HTML file because it contains the word attachments.* But obviously deleting a single file after the download is nothing to worry about.


*This is the link to the file, in case anyone is curious:

Code: 
https://titsintops.com/phpBB2/index.php?attachments/upload&type=post&context[post_id]=1463815&hash=92152f6fe0c3f79c7bd73f34b15605f9
Click to expand...
This is late but ty so much. Wouldn't have figured out this method on my own.
 
Bash: 
$ pip install -r requirements
fails with the following error message on various Linux and Mac setups I've tried recently. Perhaps due to python 3.10 update?

Bash: 
    ERROR: Command errored out with exit status 1:
    / command: /home/username/tit-thread-scraper/env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-c9344xfg/install-record.txt --single-version-externally-managed --compile --install-headers /home/username/tit-thread-scraper/env/include/site/python3.10/typed-ast
         cwd: /tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/
    Complete output (251 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-3.10
    creating build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/conversions.py -> build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/ast3.py -> build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/ast27.py -> build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/__init__.py -> build/lib.linux-armv7l-3.10/typed_ast
    package init file 'ast3/tests/__init__.py' not found (or not a regular file)
    creating build/lib.linux-armv7l-3.10/typed_ast/tests
    copying ast3/tests/test_basics.py -> build/lib.linux-armv7l-3.10/typed_ast/tests
    running build_ext
    building '_ast27' extension
    creating build/temp.linux-armv7l-3.10
    creating build/temp.linux-armv7l-3.10/ast27
    creating build/temp.linux-armv7l-3.10/ast27/Custom
    creating build/temp.linux-armv7l-3.10/ast27/Parser
    creating build/temp.linux-armv7l-3.10/ast27/Python
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O3 -pipe -fstack-protector-strong -fno-plt -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O3 -pipe -fstack-protector-strong -fno-plt -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O3 -pipe -fstack-protector-strong -fno-plt -fPIC -Iast27/Include -I/home/username/tit-thread-scraper/env/include -I/usr/include/python3.10 -c ast27/Custom/typed_ast.c -o build/temp.linux-armv7l-3.10/ast27/Custom/typed_ast.o
    In file included from ast27/Custom/../Include/Python-ast.h:3,
                     from ast27/Custom/typed_ast.c:2:
    ast27/Custom/../Include/../Include/asdl.h:32:41: error: unknown type name ‘PyArena’
       32 | asdl_seq *asdl_seq_new(Py_ssize_t size, PyArena *arena);
          |                                         ^~~~~~~
    ast27/Custom/../Include/../Include/asdl.h:33:49: error: unknown type name ‘PyArena’
       33 | asdl_int_seq *asdl_int_seq_new(Py_ssize_t size, PyArena *arena);
          |                                                 ^~~~~~~
    In file included from ast27/Custom/typed_ast.c:2:
    ast27/Custom/../Include/Python-ast.h:398:63: error: unknown type name ‘PyArena’
      398 | mod_ty _Ta27_Module(asdl_seq * body, asdl_seq * type_ignores, PyArena *arena);
          |                                                               ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:400:43: error: unknown type name ‘PyArena’
      400 | mod_ty _Ta27_Interactive(asdl_seq * body, PyArena *arena);
          |                                           ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:402:39: error: unknown type name ‘PyArena’
      402 | mod_ty _Ta27_Expression(expr_ty body, PyArena *arena);
          |                                       ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:404:65: error: unknown type name ‘PyArena’
      404 | mod_ty _Ta27_FunctionType(asdl_seq * argtypes, expr_ty returns, PyArena *arena);
          |                                                                 ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:406:37: error: unknown type name ‘PyArena’
      406 | mod_ty _Ta27_Suite(asdl_seq * body, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:409:92: error: unknown type name ‘PyArena’
      409 |                           decorator_list, string type_comment, int lineno, int col_offset, PyArena
          |                                                                                            ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:413:68: error: unknown type name ‘PyArena’
      413 |                        decorator_list, int lineno, int col_offset, PyArena *arena);
          |                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:415:65: error: unknown type name ‘PyArena’
      415 | stmt_ty _Ta27_Return(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                                 ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:417:70: error: unknown type name ‘PyArena’
      417 | stmt_ty _Ta27_Delete(asdl_seq * targets, int lineno, int col_offset, PyArena *arena);
          |                                                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:420:34: error: unknown type name ‘PyArena’
      420 |                      col_offset, PyArena *arena);
          |                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:423:25: error: unknown type name ‘PyArena’
      423 |                         PyArena *arena);
          |                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:425:91: error: unknown type name ‘PyArena’
      425 | stmt_ty _Ta27_Print(expr_ty dest, asdl_seq * values, bool nl, int lineno, int col_offset, PyArena
          |                                                                                           ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:429:61: error: unknown type name ‘PyArena’
      429 |                   type_comment, int lineno, int col_offset, PyArena *arena);
          |                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:432:21: error: unknown type name ‘PyArena’
      432 |                     PyArena *arena);
          |                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:435:18: error: unknown type name ‘PyArena’
      435 |                  PyArena *arena);
          |                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:438:62: error: unknown type name ‘PyArena’
      438 |                    type_comment, int lineno, int col_offset, PyArena *arena);
          |                                                              ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:440:92: error: unknown type name ‘PyArena’
      440 | stmt_ty _Ta27_Raise(expr_ty type, expr_ty inst, expr_ty tback, int lineno, int col_offset, PyArena
          |                                                                                            ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:444:37: error: unknown type name ‘PyArena’
      444 |                         col_offset, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:446:93: error: unknown type name ‘PyArena’
      446 | stmt_ty _Ta27_TryFinally(asdl_seq * body, asdl_seq * finalbody, int lineno, int col_offset, PyArena
          |                                                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:449:77: error: unknown type name ‘PyArena’
      449 | stmt_ty _Ta27_Assert(expr_ty test, expr_ty msg, int lineno, int col_offset, PyArena *arena);
          |                                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:451:68: error: unknown type name ‘PyArena’
      451 | stmt_ty _Ta27_Import(asdl_seq * names, int lineno, int col_offset, PyArena *arena);
          |                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:454:38: error: unknown type name ‘PyArena’
      454 |                          col_offset, PyArena *arena);
          |                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:457:20: error: unknown type name ‘PyArena’
      457 |                    PyArena *arena);
          |                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:459:68: error: unknown type name ‘PyArena’
      459 | stmt_ty _Ta27_Global(asdl_seq * names, int lineno, int col_offset, PyArena *arena);
          |                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:461:63: error: unknown type name ‘PyArena’
      461 | stmt_ty _Ta27_Expr(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                               ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:463:48: error: unknown type name ‘PyArena’
      463 | stmt_ty _Ta27_Pass(int lineno, int col_offset, PyArena *arena);
          |                                                ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:465:49: error: unknown type name ‘PyArena’
      465 | stmt_ty _Ta27_Break(int lineno, int col_offset, PyArena *arena);
          |                                                 ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:467:52: error: unknown type name ‘PyArena’
      467 | stmt_ty _Ta27_Continue(int lineno, int col_offset, PyArena *arena);
          |                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:469:83: error: unknown type name ‘PyArena’
      469 | expr_ty _Ta27_BoolOp(boolop_ty op, asdl_seq * values, int lineno, int col_offset, PyArena *arena);
          |                                                                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:472:21: error: unknown type name ‘PyArena’
      472 |                     PyArena *arena);
          |                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:474:83: error: unknown type name ‘PyArena’
      474 | expr_ty _Ta27_UnaryOp(unaryop_ty op, expr_ty operand, int lineno, int col_offset, PyArena *arena);
          |                                                                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:476:83: error: unknown type name ‘PyArena’
      476 | expr_ty _Ta27_Lambda(arguments_ty args, expr_ty body, int lineno, int col_offset, PyArena *arena);
          |                                                                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:478:93: error: unknown type name ‘PyArena’
      478 | expr_ty _Ta27_IfExp(expr_ty test, expr_ty body, expr_ty orelse, int lineno, int col_offset, PyArena
          |                                                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:481:84: error: unknown type name ‘PyArena’
      481 | expr_ty _Ta27_Dict(asdl_seq * keys, asdl_seq * values, int lineno, int col_offset, PyArena *arena);
          |                                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:483:64: error: unknown type name ‘PyArena’
      483 | expr_ty _Ta27_Set(asdl_seq * elts, int lineno, int col_offset, PyArena *arena);
          |                                                                ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:485:88: error: unknown type name ‘PyArena’
      485 | expr_ty _Ta27_ListComp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena
          |                                                                                        ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:488:87: error: unknown type name ‘PyArena’
      488 | expr_ty _Ta27_SetComp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena
          |                                                                                       ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:492:36: error: unknown type name ‘PyArena’
      492 |                        col_offset, PyArena *arena);
          |                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:494:92: error: unknown type name ‘PyArena’
      494 | expr_ty _Ta27_GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena
          |                                                                                            ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:497:64: error: unknown type name ‘PyArena’
      497 | expr_ty _Ta27_Yield(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                                ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:500:35: error: unknown type name ‘PyArena’
      500 |                       col_offset, PyArena *arena);
          |                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:503:56: error: unknown type name ‘PyArena’
      503 |                    kwargs, int lineno, int col_offset, PyArena *arena);
          |                                                        ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:505:63: error: unknown type name ‘PyArena’
      505 | expr_ty _Ta27_Repr(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                               ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:507:57: error: unknown type name ‘PyArena’
      507 | expr_ty _Ta27_Num(object n, int lineno, int col_offset, PyArena *arena);
          |                                                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:509:70: error: unknown type name ‘PyArena’
      509 | expr_ty _Ta27_Str(string s, string kind, int lineno, int col_offset, PyArena *arena);
          |                                                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:512:37: error: unknown type name ‘PyArena’
      512 |                         col_offset, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:515:37: error: unknown type name ‘PyArena’
      515 |                         col_offset, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:517:84: error: unknown type name ‘PyArena’
      517 | expr_ty _Ta27_Name(identifier id, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena);
          |                                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:519:86: error: unknown type name ‘PyArena’
      519 | expr_ty _Ta27_List(asdl_seq * elts, expr_context_ty ctx, int lineno, int col_offset, PyArena
          |                                                                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:522:87: error: unknown type name ‘PyArena’
      522 | expr_ty _Ta27_Tuple(asdl_seq * elts, expr_context_ty ctx, int lineno, int col_offset, PyArena
          |                                                                                       ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:525:25: error: unknown type name ‘PyArena’
      525 | slice_ty _Ta27_Ellipsis(PyArena *arena);
          |                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:527:66: error: unknown type name ‘PyArena’
      527 | slice_ty _Ta27_Slice(expr_ty lower, expr_ty upper, expr_ty step, PyArena *arena);
          |                                                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:529:42: error: unknown type name ‘PyArena’
      529 | slice_ty _Ta27_ExtSlice(asdl_seq * dims, PyArena *arena);
          |                                          ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:531:37: error: unknown type name ‘PyArena’
      531 | slice_ty _Ta27_Index(expr_ty value, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:533:84: error: unknown type name ‘PyArena’
      533 | comprehension_ty _Ta27_comprehension(expr_ty target, expr_ty iter, asdl_seq * ifs, PyArena *arena);
          |                                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:536:50: error: unknown type name ‘PyArena’
      536 |                                      col_offset, PyArena *arena);
          |                                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:539:66: error: unknown type name ‘PyArena’
      539 |                              defaults, asdl_seq * type_comments, PyArena *arena);
          |                                                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:541:57: error: unknown type name ‘PyArena’
      541 | keyword_ty _Ta27_keyword(identifier arg, expr_ty value, PyArena *arena);
          |                                                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:543:58: error: unknown type name ‘PyArena’
      543 | alias_ty _Ta27_alias(identifier name, identifier asname, PyArena *arena);
          |                                                          ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:545:57: error: unknown type name ‘PyArena’
      545 | type_ignore_ty _Ta27_TypeIgnore(int lineno, string tag, PyArena *arena);
          |                                                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:548:39: error: unknown type name ‘PyArena’
      548 | mod_ty Ta27AST_obj2mod(PyObject* ast, PyArena* arena, int mode);
          |                                       ^~~~~~~
    In file included from ast27/Custom/typed_ast.c:7:
    ast27/Custom/../Include/ast.h:8:21: error: unknown type name ‘PyArena’
        8 |       const char *, PyArena *);
          |                     ^~~~~~~
    ast27/Custom/typed_ast.c:210:54: error: unknown type name ‘PyArena’
      210 |                              PyCompilerFlags *flags, PyArena *arena)
          |                                                      ^~~~~~~
    ast27/Custom/typed_ast.c: In function ‘string_object_to_py_ast’:
    ast27/Custom/typed_ast.c:244:5: error: unknown type name ‘PyArena’
      244 |     PyArena *arena = PyArena_New();
          |     ^~~~~~~
    ast27/Custom/typed_ast.c:244:22: warning: implicit declaration of function ‘PyArena_New’; did you mean ‘PyGen_New’? [-Wimplicit-function-declaration]
      244 |     PyArena *arena = PyArena_New();
          |                      ^~~~~~~~~~~
          |                      PyGen_New
    ast27/Custom/typed_ast.c:244:22: warning: initialization of ‘int *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
    ast27/Custom/typed_ast.c:248:11: warning: implicit declaration of function ‘string_object_to_c_ast’; did you mean ‘string_object_to_py_ast’? [-Wimplicit-function-declaration]
      248 |     mod = string_object_to_c_ast(str, filename, start, flags, arena);
          |           ^~~~~~~~~~~~~~~~~~~~~~
          |           string_object_to_py_ast
    ast27/Custom/typed_ast.c:248:9: warning: assignment to ‘mod_ty’ {aka ‘struct _mod *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
      248 |     mod = string_object_to_c_ast(str, filename, start, flags, arena);
          |         ^
    ast27/Custom/typed_ast.c:250:9: warning: implicit declaration of function ‘PyArena_Free’; did you mean ‘PyMem_Free’? [-Wimplicit-function-declaration]
      250 |         PyArena_Free(arena);
          |         ^~~~~~~~~~~~
          |         PyMem_Free
    At top level:
    ast27/Custom/typed_ast.c:201:1: warning: ‘err_free’ defined but not used [-Wunused-function]
      201 | err_free(perrdetail *err)
          | ^~~~~~~~
    ast27/Custom/typed_ast.c:81:1: warning: ‘err_input’ defined but not used [-Wunused-function]
       81 | err_input(perrdetail *err)
          | ^~~~~~~~~
    ast27/Custom/typed_ast.c:66:12: warning: ‘PARSER_FLAGS’ defined but not used [-Wunused-function]
       66 | static int PARSER_FLAGS(PyCompilerFlags *flags)
          |            ^~~~~~~~~~~~
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/username/tit-thread-scraper/env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-c9344xfg/install-record.txt --single-version-externally-managed --compile --install-headers /home/username/tit-thread-scraper/env/include/site/python3.10/typed-ast Check the logs for full command output.
 
renstadlt said:
Bash: 
$ pip install -r requirements
fails with the following error message on various Linux and Mac setups I've tried recently. Perhaps due to python 3.10 update?

Bash: 
    ERROR: Command errored out with exit status 1:
    / command: /home/username/tit-thread-scraper/env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-c9344xfg/install-record.txt --single-version-externally-managed --compile --install-headers /home/username/tit-thread-scraper/env/include/site/python3.10/typed-ast
         cwd: /tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/
    Complete output (251 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-3.10
    creating build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/conversions.py -> build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/ast3.py -> build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/ast27.py -> build/lib.linux-armv7l-3.10/typed_ast
    copying typed_ast/__init__.py -> build/lib.linux-armv7l-3.10/typed_ast
    package init file 'ast3/tests/__init__.py' not found (or not a regular file)
    creating build/lib.linux-armv7l-3.10/typed_ast/tests
    copying ast3/tests/test_basics.py -> build/lib.linux-armv7l-3.10/typed_ast/tests
    running build_ext
    building '_ast27' extension
    creating build/temp.linux-armv7l-3.10
    creating build/temp.linux-armv7l-3.10/ast27
    creating build/temp.linux-armv7l-3.10/ast27/Custom
    creating build/temp.linux-armv7l-3.10/ast27/Parser
    creating build/temp.linux-armv7l-3.10/ast27/Python
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O3 -pipe -fstack-protector-strong -fno-plt -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O3 -pipe -fstack-protector-strong -fno-plt -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O3 -pipe -fstack-protector-strong -fno-plt -fPIC -Iast27/Include -I/home/username/tit-thread-scraper/env/include -I/usr/include/python3.10 -c ast27/Custom/typed_ast.c -o build/temp.linux-armv7l-3.10/ast27/Custom/typed_ast.o
    In file included from ast27/Custom/../Include/Python-ast.h:3,
                     from ast27/Custom/typed_ast.c:2:
    ast27/Custom/../Include/../Include/asdl.h:32:41: error: unknown type name ‘PyArena’
       32 | asdl_seq *asdl_seq_new(Py_ssize_t size, PyArena *arena);
          |                                         ^~~~~~~
    ast27/Custom/../Include/../Include/asdl.h:33:49: error: unknown type name ‘PyArena’
       33 | asdl_int_seq *asdl_int_seq_new(Py_ssize_t size, PyArena *arena);
          |                                                 ^~~~~~~
    In file included from ast27/Custom/typed_ast.c:2:
    ast27/Custom/../Include/Python-ast.h:398:63: error: unknown type name ‘PyArena’
      398 | mod_ty _Ta27_Module(asdl_seq * body, asdl_seq * type_ignores, PyArena *arena);
          |                                                               ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:400:43: error: unknown type name ‘PyArena’
      400 | mod_ty _Ta27_Interactive(asdl_seq * body, PyArena *arena);
          |                                           ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:402:39: error: unknown type name ‘PyArena’
      402 | mod_ty _Ta27_Expression(expr_ty body, PyArena *arena);
          |                                       ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:404:65: error: unknown type name ‘PyArena’
      404 | mod_ty _Ta27_FunctionType(asdl_seq * argtypes, expr_ty returns, PyArena *arena);
          |                                                                 ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:406:37: error: unknown type name ‘PyArena’
      406 | mod_ty _Ta27_Suite(asdl_seq * body, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:409:92: error: unknown type name ‘PyArena’
      409 |                           decorator_list, string type_comment, int lineno, int col_offset, PyArena
          |                                                                                            ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:413:68: error: unknown type name ‘PyArena’
      413 |                        decorator_list, int lineno, int col_offset, PyArena *arena);
          |                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:415:65: error: unknown type name ‘PyArena’
      415 | stmt_ty _Ta27_Return(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                                 ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:417:70: error: unknown type name ‘PyArena’
      417 | stmt_ty _Ta27_Delete(asdl_seq * targets, int lineno, int col_offset, PyArena *arena);
          |                                                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:420:34: error: unknown type name ‘PyArena’
      420 |                      col_offset, PyArena *arena);
          |                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:423:25: error: unknown type name ‘PyArena’
      423 |                         PyArena *arena);
          |                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:425:91: error: unknown type name ‘PyArena’
      425 | stmt_ty _Ta27_Print(expr_ty dest, asdl_seq * values, bool nl, int lineno, int col_offset, PyArena
          |                                                                                           ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:429:61: error: unknown type name ‘PyArena’
      429 |                   type_comment, int lineno, int col_offset, PyArena *arena);
          |                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:432:21: error: unknown type name ‘PyArena’
      432 |                     PyArena *arena);
          |                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:435:18: error: unknown type name ‘PyArena’
      435 |                  PyArena *arena);
          |                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:438:62: error: unknown type name ‘PyArena’
      438 |                    type_comment, int lineno, int col_offset, PyArena *arena);
          |                                                              ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:440:92: error: unknown type name ‘PyArena’
      440 | stmt_ty _Ta27_Raise(expr_ty type, expr_ty inst, expr_ty tback, int lineno, int col_offset, PyArena
          |                                                                                            ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:444:37: error: unknown type name ‘PyArena’
      444 |                         col_offset, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:446:93: error: unknown type name ‘PyArena’
      446 | stmt_ty _Ta27_TryFinally(asdl_seq * body, asdl_seq * finalbody, int lineno, int col_offset, PyArena
          |                                                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:449:77: error: unknown type name ‘PyArena’
      449 | stmt_ty _Ta27_Assert(expr_ty test, expr_ty msg, int lineno, int col_offset, PyArena *arena);
          |                                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:451:68: error: unknown type name ‘PyArena’
      451 | stmt_ty _Ta27_Import(asdl_seq * names, int lineno, int col_offset, PyArena *arena);
          |                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:454:38: error: unknown type name ‘PyArena’
      454 |                          col_offset, PyArena *arena);
          |                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:457:20: error: unknown type name ‘PyArena’
      457 |                    PyArena *arena);
          |                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:459:68: error: unknown type name ‘PyArena’
      459 | stmt_ty _Ta27_Global(asdl_seq * names, int lineno, int col_offset, PyArena *arena);
          |                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:461:63: error: unknown type name ‘PyArena’
      461 | stmt_ty _Ta27_Expr(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                               ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:463:48: error: unknown type name ‘PyArena’
      463 | stmt_ty _Ta27_Pass(int lineno, int col_offset, PyArena *arena);
          |                                                ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:465:49: error: unknown type name ‘PyArena’
      465 | stmt_ty _Ta27_Break(int lineno, int col_offset, PyArena *arena);
          |                                                 ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:467:52: error: unknown type name ‘PyArena’
      467 | stmt_ty _Ta27_Continue(int lineno, int col_offset, PyArena *arena);
          |                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:469:83: error: unknown type name ‘PyArena’
      469 | expr_ty _Ta27_BoolOp(boolop_ty op, asdl_seq * values, int lineno, int col_offset, PyArena *arena);
          |                                                                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:472:21: error: unknown type name ‘PyArena’
      472 |                     PyArena *arena);
          |                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:474:83: error: unknown type name ‘PyArena’
      474 | expr_ty _Ta27_UnaryOp(unaryop_ty op, expr_ty operand, int lineno, int col_offset, PyArena *arena);
          |                                                                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:476:83: error: unknown type name ‘PyArena’
      476 | expr_ty _Ta27_Lambda(arguments_ty args, expr_ty body, int lineno, int col_offset, PyArena *arena);
          |                                                                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:478:93: error: unknown type name ‘PyArena’
      478 | expr_ty _Ta27_IfExp(expr_ty test, expr_ty body, expr_ty orelse, int lineno, int col_offset, PyArena
          |                                                                                             ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:481:84: error: unknown type name ‘PyArena’
      481 | expr_ty _Ta27_Dict(asdl_seq * keys, asdl_seq * values, int lineno, int col_offset, PyArena *arena);
          |                                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:483:64: error: unknown type name ‘PyArena’
      483 | expr_ty _Ta27_Set(asdl_seq * elts, int lineno, int col_offset, PyArena *arena);
          |                                                                ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:485:88: error: unknown type name ‘PyArena’
      485 | expr_ty _Ta27_ListComp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena
          |                                                                                        ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:488:87: error: unknown type name ‘PyArena’
      488 | expr_ty _Ta27_SetComp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena
          |                                                                                       ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:492:36: error: unknown type name ‘PyArena’
      492 |                        col_offset, PyArena *arena);
          |                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:494:92: error: unknown type name ‘PyArena’
      494 | expr_ty _Ta27_GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena
          |                                                                                            ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:497:64: error: unknown type name ‘PyArena’
      497 | expr_ty _Ta27_Yield(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                                ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:500:35: error: unknown type name ‘PyArena’
      500 |                       col_offset, PyArena *arena);
          |                                   ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:503:56: error: unknown type name ‘PyArena’
      503 |                    kwargs, int lineno, int col_offset, PyArena *arena);
          |                                                        ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:505:63: error: unknown type name ‘PyArena’
      505 | expr_ty _Ta27_Repr(expr_ty value, int lineno, int col_offset, PyArena *arena);
          |                                                               ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:507:57: error: unknown type name ‘PyArena’
      507 | expr_ty _Ta27_Num(object n, int lineno, int col_offset, PyArena *arena);
          |                                                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:509:70: error: unknown type name ‘PyArena’
      509 | expr_ty _Ta27_Str(string s, string kind, int lineno, int col_offset, PyArena *arena);
          |                                                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:512:37: error: unknown type name ‘PyArena’
      512 |                         col_offset, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:515:37: error: unknown type name ‘PyArena’
      515 |                         col_offset, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:517:84: error: unknown type name ‘PyArena’
      517 | expr_ty _Ta27_Name(identifier id, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena);
          |                                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:519:86: error: unknown type name ‘PyArena’
      519 | expr_ty _Ta27_List(asdl_seq * elts, expr_context_ty ctx, int lineno, int col_offset, PyArena
          |                                                                                      ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:522:87: error: unknown type name ‘PyArena’
      522 | expr_ty _Ta27_Tuple(asdl_seq * elts, expr_context_ty ctx, int lineno, int col_offset, PyArena
          |                                                                                       ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:525:25: error: unknown type name ‘PyArena’
      525 | slice_ty _Ta27_Ellipsis(PyArena *arena);
          |                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:527:66: error: unknown type name ‘PyArena’
      527 | slice_ty _Ta27_Slice(expr_ty lower, expr_ty upper, expr_ty step, PyArena *arena);
          |                                                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:529:42: error: unknown type name ‘PyArena’
      529 | slice_ty _Ta27_ExtSlice(asdl_seq * dims, PyArena *arena);
          |                                          ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:531:37: error: unknown type name ‘PyArena’
      531 | slice_ty _Ta27_Index(expr_ty value, PyArena *arena);
          |                                     ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:533:84: error: unknown type name ‘PyArena’
      533 | comprehension_ty _Ta27_comprehension(expr_ty target, expr_ty iter, asdl_seq * ifs, PyArena *arena);
          |                                                                                    ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:536:50: error: unknown type name ‘PyArena’
      536 |                                      col_offset, PyArena *arena);
          |                                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:539:66: error: unknown type name ‘PyArena’
      539 |                              defaults, asdl_seq * type_comments, PyArena *arena);
          |                                                                  ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:541:57: error: unknown type name ‘PyArena’
      541 | keyword_ty _Ta27_keyword(identifier arg, expr_ty value, PyArena *arena);
          |                                                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:543:58: error: unknown type name ‘PyArena’
      543 | alias_ty _Ta27_alias(identifier name, identifier asname, PyArena *arena);
          |                                                          ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:545:57: error: unknown type name ‘PyArena’
      545 | type_ignore_ty _Ta27_TypeIgnore(int lineno, string tag, PyArena *arena);
          |                                                         ^~~~~~~
    ast27/Custom/../Include/Python-ast.h:548:39: error: unknown type name ‘PyArena’
      548 | mod_ty Ta27AST_obj2mod(PyObject* ast, PyArena* arena, int mode);
          |                                       ^~~~~~~
    In file included from ast27/Custom/typed_ast.c:7:
    ast27/Custom/../Include/ast.h:8:21: error: unknown type name ‘PyArena’
        8 |       const char *, PyArena *);
          |                     ^~~~~~~
    ast27/Custom/typed_ast.c:210:54: error: unknown type name ‘PyArena’
      210 |                              PyCompilerFlags *flags, PyArena *arena)
          |                                                      ^~~~~~~
    ast27/Custom/typed_ast.c: In function ‘string_object_to_py_ast’:
    ast27/Custom/typed_ast.c:244:5: error: unknown type name ‘PyArena’
      244 |     PyArena *arena = PyArena_New();
          |     ^~~~~~~
    ast27/Custom/typed_ast.c:244:22: warning: implicit declaration of function ‘PyArena_New’; did you mean ‘PyGen_New’? [-Wimplicit-function-declaration]
      244 |     PyArena *arena = PyArena_New();
          |                      ^~~~~~~~~~~
          |                      PyGen_New
    ast27/Custom/typed_ast.c:244:22: warning: initialization of ‘int *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
    ast27/Custom/typed_ast.c:248:11: warning: implicit declaration of function ‘string_object_to_c_ast’; did you mean ‘string_object_to_py_ast’? [-Wimplicit-function-declaration]
      248 |     mod = string_object_to_c_ast(str, filename, start, flags, arena);
          |           ^~~~~~~~~~~~~~~~~~~~~~
          |           string_object_to_py_ast
    ast27/Custom/typed_ast.c:248:9: warning: assignment to ‘mod_ty’ {aka ‘struct _mod *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
      248 |     mod = string_object_to_c_ast(str, filename, start, flags, arena);
          |         ^
    ast27/Custom/typed_ast.c:250:9: warning: implicit declaration of function ‘PyArena_Free’; did you mean ‘PyMem_Free’? [-Wimplicit-function-declaration]
      250 |         PyArena_Free(arena);
          |         ^~~~~~~~~~~~
          |         PyMem_Free
    At top level:
    ast27/Custom/typed_ast.c:201:1: warning: ‘err_free’ defined but not used [-Wunused-function]
      201 | err_free(perrdetail *err)
          | ^~~~~~~~
    ast27/Custom/typed_ast.c:81:1: warning: ‘err_input’ defined but not used [-Wunused-function]
       81 | err_input(perrdetail *err)
          | ^~~~~~~~~
    ast27/Custom/typed_ast.c:66:12: warning: ‘PARSER_FLAGS’ defined but not used [-Wunused-function]
       66 | static int PARSER_FLAGS(PyCompilerFlags *flags)
          |            ^~~~~~~~~~~~
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/username/tit-thread-scraper/env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kqhnytow/typed-ast_fd216cce4c604423ae10d2e6edad68f9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-c9344xfg/install-record.txt --single-version-externally-managed --compile --install-headers /home/username/tit-thread-scraper/env/include/site/python3.10/typed-ast Check the logs for full command output.
Click to expand...
Pretty weird. I just set up a fresh venv and installed, no issues. Maybe your Python install is fucked. You could try:
  • Reinstall python and python-pip
  • Delete and regenerate your venv
  • Run pip install --upgrade pip from the venv
I also notice that setup.py is compiling ast3 and ast27, which is odd. If the steps above don't work, then it'll be googling time, because this seems like either a python installation issue or a packaging issue.
 
Thanks for trying to reproduce the issue. I ended up spinning up a linux VM and installing the dependencies globally as distribution packages and everything worked fine.
 
Thanks for this. I had the same issue as above but manually installing bs4 via `pip install bs4` worked

Time to fill up my Mega archive
 
Appreciate the hard work you have put into this script.

Might I suggest putting it on GitHub? That way, people can always download the latest release, keep up with the development, and possibly even submit some PRs :cool:
 
LulzGuy said:
Appreciate the hard work you have put into this script.

Might I suggest putting it on GitHub? That way, people can always download the latest release, keep up with the development, and possibly even submit some PRs :cool:
Click to expand...
Yeah, thought about it, but I try to keep my porn life as isolated as possible, and didn't feel like making a dummy account. If anyone wants to take this and host it publicly, feel free. Consider the code public domain.

I also considered generalizing the code, since I've seen other forums that look exactly the same, so this site is probably a mostly-default install of some php forum backend. But, I'm too lazy.

Speaking of, I've added pagination for file saving. IE, page 1's media will be saved into a folder '0001', and page 470 will save into a folder '0470'. I've found this helps performance-wise on those big threads like Milena Velba, where 30,000 files in one directory starts to slow things down considerably (at least on my zfs/cifs setup). To use it, call the script with the -g or --paginate argument.
 

Attachments

  • tit-thread-scraper.zip
    5.8 KB · Views: 275
You must log in or register to reply here.

Similar threads

P
Discussing the Tits In Tops Hall Of Fame thread
Replies
16
Views
2,014
Nevermore
Nevermore
P
Lima Beans: Super Busty Facebook Reels of Tits In Tops
Replies
1
Views
844
Tscott7
T
Zweii
Can't reach Tits in Tops with Brave browser
Replies
1
Views
239
Alacres
A
blabba75
Boob Curtains / Loose crop tops draped over big tits
Replies
3
Views
744
bknewbie
B
8
Metallic tits in tops
Replies
1
Views
393
Chiuaua223
Chiuaua223
Top