b47bd9ed56
moved db conn variable to db.go
2021-02-13 02:48:11 +00:00
5a481121d2
updated indexes yet again
2021-02-13 02:46:32 +00:00
d4ef1561d4
log messages moved to logConn
2021-02-13 02:45:24 +00:00
163dee0389
removed dead code
2021-02-12 02:14:20 +00:00
cefacf7462
Status update line, disabling web server
2021-02-12 00:26:31 +00:00
947645828e
adding LogLevel setting
2021-02-11 21:05:01 +00:00
d6b03c5254
cleanup, added go mod files, table indexing properly
2021-02-11 12:34:49 +00:00
778150f83c
Rearranging files
...
Down the line I need to move headers.go and uniquefifo.go into a module
2021-02-10 23:58:20 +00:00
126aac8c81
decoupling retrieval of actor from activity in database
2021-02-10 22:30:34 +00:00
6efd723aac
Updating table index and tsvector creation
2021-02-03 01:28:27 +00:00
9dabd87f06
forgot to use commit uniquefifo.go
2021-02-03 00:07:51 +00:00
a122f72af7
Fully migrated to jsonb, adjusted uniquefifo to keep cache fresh
...
* Added fifo mechanism to actors table
* Increased fifo size to 10
* Still getting some database insert duplicate errors, but only for
very active instances when they are newly identified
2021-02-02 23:34:43 +00:00
f262de1dc3
migrating to storing data as jsonb object in database
...
captures all data, avoids cache-misses
2021-02-01 20:31:40 +00:00
3662535b0d
renaming posts to activities and accounts to actors
...
removed old posthash mechanism
2021-02-01 12:52:42 +00:00
102ddbe41c
Adding recent uris struct to manually added instances
2021-02-01 02:05:56 +00:00
7e71d0cc7a
Installed recent requests structure to prevent repeat requests
...
Default size is 5
Only targets posts, not users
2021-02-01 00:28:20 +00:00
88d058528f
Changed logging to stdout
2021-02-01 00:24:35 +00:00
55cb51b94e
added or adjusted missing closes
2021-01-30 14:30:49 +00:00
cd8ecce807
Added proxy support
...
Some IPv6 traffic still sending directly to instance
not sure why not
2021-01-30 07:12:37 +00:00
2f9e0f85e4
Tor is on port 9050, Proxy commented out by default
2021-01-30 07:10:49 +00:00
1f3cbe8bde
Isolated BuildClient from GetRunner
...
Now there is a single place where new clients are built
2021-01-30 05:54:03 +00:00
96e47f0373
Removing GetHTTPSession for GetRunner
2021-01-29 17:44:16 -05:00
4706bce7e3
hardcoded user-agent as tusky + added a DoTries()
2021-01-25 21:06:47 -05:00
1bebd9064c
removing urls from normalization
2021-01-25 20:28:13 -05:00
1203d4f164
added log.go
2021-01-17 04:17:15 +00:00
9d2601e6c9
Replaces <br> tags with spaces
2021-01-14 20:43:20 +00:00
1adaba8322
retrying connections + logging prefix
...
this is because some hosting providers throttle rapid new connections
regex additions
2021-01-14 19:51:42 +00:00
e9bd9b67cf
minor regex checking
2021-01-10 21:47:08 +00:00
ec1743905a
validating hostname before starting instance
...
changed hostname identification from account to URI
2021-01-10 05:31:51 +00:00
48acaa5b0b
updated table constraints
2021-01-06 02:43:41 +00:00
5ea270f9ec
added table relationship
2021-01-05 05:54:27 +00:00
81b06d64e8
hack around ignoring 'to' URIs that end in /followers
...
may remove the entire associated for-loop...
2021-01-04 18:40:11 +00:00
d02ae808a1
reducing potential keep-alive connections to 1 per hostname (not per IP)
2021-01-02 07:00:21 +00:00
7973cc9600
go report card
2020-12-29 20:21:38 +00:00
85e0735fa3
go fmt'ed
2020-12-29 20:20:02 +00:00
b2974b7501
added crawling for web reciever
...
fixed crawling settings condition
2020-12-29 20:06:18 +00:00
20fb4ed76a
mostly web, fixed a regression
2020-12-29 17:30:26 +00:00
5914fc0890
added stream retry, probably a bad approach
2020-12-29 16:41:43 +00:00
5473052519
changed comments, made streaming in a loop
2020-12-29 16:38:14 +00:00
72821c4641
Banned sinblr.com
2020-12-29 16:33:14 +00:00
d50dc1ec0d
Cleaning up some errors
2020-12-29 15:47:49 +00:00
c93dd89332
blocking twitiverse.com by default
2020-12-29 15:05:57 +00:00
63bc88324b
changed to pool from connection, few error fixes
2020-12-29 15:03:52 +00:00
eafb3b9318
added normalization
2020-12-25 05:45:08 +00:00
88c074f76b
reduced the number of new connections to 1, reduce spamming servers
...
set keep-alive to 2 hours
probably a bunch of regressions but I don't have unit tests yet
2020-12-25 05:18:42 +00:00
9db3c05ed6
few bugs
2020-12-23 08:20:03 +00:00
777120518a
removed dead code + channel
...
ctl is broken
2020-12-22 20:36:37 +00:00
28fa8ab5ec
deleting comments
2020-12-22 20:30:21 +00:00
39b53ed45c
added stream and poll
...
fixed tables.sql
2020-12-22 20:20:12 +00:00
6b8d6a3bef
reusing connections
2020-12-22 19:46:34 +00:00
