|
|
7e71d0cc7a
|
Installed recent requests structure to prevent repeat requests
Default size is 5
Only targets posts, not users
|
2021-02-01 00:28:20 +00:00 |
|
|
|
88d058528f
|
Changed logging to stdout
|
2021-02-01 00:24:35 +00:00 |
|
|
|
55cb51b94e
|
added or adjusted missing closes
|
2021-01-30 14:30:49 +00:00 |
|
|
|
cd8ecce807
|
Added proxy support
Some IPv6 traffic still sending directly to instance
not sure why not
|
2021-01-30 07:12:37 +00:00 |
|
|
|
2f9e0f85e4
|
Tor is on port 9050, Proxy commented out by default
|
2021-01-30 07:10:49 +00:00 |
|
|
|
1f3cbe8bde
|
Isolated BuildClient from GetRunner
Now there is a single place where new clients are built
|
2021-01-30 05:54:03 +00:00 |
|
|
|
96e47f0373
|
Removing GetHTTPSession for GetRunner
|
2021-01-29 17:44:16 -05:00 |
|
|
|
4706bce7e3
|
hardcoded user-agent as tusky + added a DoTries()
|
2021-01-25 21:06:47 -05:00 |
|
|
|
1bebd9064c
|
removing urls from normalization
|
2021-01-25 20:28:13 -05:00 |
|
|
|
1203d4f164
|
added log.go
|
2021-01-17 04:17:15 +00:00 |
|
|
|
9d2601e6c9
|
Replaces <br> tags with spaces
|
2021-01-14 20:43:20 +00:00 |
|
|
|
1adaba8322
|
retrying connections + logging prefix
this is because some hosting providers throttle rapid new connections
regex additions
|
2021-01-14 19:51:42 +00:00 |
|
|
|
e9bd9b67cf
|
minor regex checking
|
2021-01-10 21:47:08 +00:00 |
|
|
|
ec1743905a
|
validating hostname before starting instance
changed hostname identification from account to URI
|
2021-01-10 05:31:51 +00:00 |
|
|
|
48acaa5b0b
|
updated table constraints
|
2021-01-06 02:43:41 +00:00 |
|
|
|
5ea270f9ec
|
added table relationship
|
2021-01-05 05:54:27 +00:00 |
|
|
|
81b06d64e8
|
hack around ignoring 'to' URIs that end in /followers
may remove the entire associated for-loop...
|
2021-01-04 18:40:11 +00:00 |
|
|
|
d02ae808a1
|
reducing potential keep-alive connections to 1 per hostname (not per IP)
|
2021-01-02 07:00:21 +00:00 |
|
|
|
7973cc9600
|
go report card
|
2020-12-29 20:21:38 +00:00 |
|
|
|
85e0735fa3
|
go fmt'ed
|
2020-12-29 20:20:02 +00:00 |
|
|
|
b2974b7501
|
added crawling for web reciever
fixed crawling settings condition
|
2020-12-29 20:06:18 +00:00 |
|
|
|
20fb4ed76a
|
mostly web, fixed a regression
|
2020-12-29 17:30:26 +00:00 |
|
|
|
5914fc0890
|
added stream retry, probably a bad approach
|
2020-12-29 16:41:43 +00:00 |
|
|
|
5473052519
|
changed comments, made streaming in a loop
|
2020-12-29 16:38:14 +00:00 |
|
|
|
72821c4641
|
Banned sinblr.com
|
2020-12-29 16:33:14 +00:00 |
|
|
|
d50dc1ec0d
|
Cleaning up some errors
|
2020-12-29 15:47:49 +00:00 |
|
|
|
c93dd89332
|
blocking twitiverse.com by default
|
2020-12-29 15:05:57 +00:00 |
|
|
|
63bc88324b
|
changed to pool from connection, few error fixes
|
2020-12-29 15:03:52 +00:00 |
|
|
|
eafb3b9318
|
added normalization
|
2020-12-25 05:45:08 +00:00 |
|
|
|
88c074f76b
|
reduced the number of new connections to 1, reduce spamming servers
set keep-alive to 2 hours
probably a bunch of regressions but I don't have unit tests yet
|
2020-12-25 05:18:42 +00:00 |
|
|
|
9db3c05ed6
|
few bugs
|
2020-12-23 08:20:03 +00:00 |
|
|
|
777120518a
|
removed dead code + channel
ctl is broken
|
2020-12-22 20:36:37 +00:00 |
|
|
|
28fa8ab5ec
|
deleting comments
|
2020-12-22 20:30:21 +00:00 |
|
|
|
39b53ed45c
|
added stream and poll
fixed tables.sql
|
2020-12-22 20:20:12 +00:00 |
|
|
|
6b8d6a3bef
|
reusing connections
|
2020-12-22 19:46:34 +00:00 |
|
|
|
e9178b1d09
|
verifying is being done
middle layer commit
database middle layer might be redundant
screwed up tables.sql
isn't re-useing connections
|
2020-12-22 17:35:05 +00:00 |
|
|
|
ad9fe9fad2
|
checking user and made 'pool' global
|
2020-12-18 06:06:32 +00:00 |
|
|
|
5bc9a68d53
|
web feature fetching user icon
adding database pool to all functions to verify before fetching
|
2020-12-17 18:26:55 +00:00 |
|
|
|
726ac7e0bd
|
formatting and linting
|
2020-12-17 04:23:25 +00:00 |
|
|
|
8d56747f1c
|
Updating Makefile and fedictl sharing same header file
|
2020-12-17 01:43:00 +00:00 |
|
|
|
20bb0ef788
|
Added license to README
|
2020-12-17 01:39:57 +00:00 |
|
|
|
394270b1d3
|
minor edit to readme
|
2020-12-17 01:38:59 +00:00 |
|
|
|
cc020bb6f0
|
Merged the two directories, no need to keep them separate
|
2020-12-17 01:37:34 +00:00 |
|
|
|
064acdcf2d
|
client added to runninginstances, can be shared across goroutines
adding basic post verification scaffolding
|
2020-12-17 01:15:17 +00:00 |
|
|
|
6c6d23a340
|
minor edit of README.md
|
2020-12-17 01:14:06 +00:00 |
|
|
|
7d4009cf8e
|
Added README.md and renamed LICENSE to all caps
|
2020-12-17 01:12:26 +00:00 |
|
|
|
dc55070065
|
Renaming the sample configuration file to avoid accidental commits
|
2020-12-16 19:10:54 +00:00 |
|
|
|
f8aa43ec4d
|
working through post and account verification
|
2020-12-16 06:21:18 +00:00 |
|
|
|
a50916eb36
|
using the same http client, adding it to running instances
some web work
lost track of what's going on and I'm not accountable to anyone...so screw it
|
2020-12-16 02:41:45 +00:00 |
|
|
|
3d196ccf41
|
Added multiple post handlers + exiting the main goroutine
|
2020-12-14 23:09:51 +00:00 |
|
