package main import ( "encoding/json" "io/ioutil" "net/http" "strings" "time" "net" ) func DoTries(o *RunningInstance, req *http.Request) (*http.Response, error) { var resp *http.Response var err error for tries := 0; tries < 10; tries++ { resp, err = o.client.Do(req) if err != nil { // URL.Scheme, Host, Path Opaque logWarn.Print("Failure connecting to " + req.URL.Scheme + "://" + req.URL.Host + req.URL.Path + ", attempt ", tries + 1, ", sleeping for 5 minutes: ", err) time.Sleep(time.Minute * 5) continue } break } return resp, err } func GetRunner(endpoint string) RunningInstance { ri_mutex.Lock() o, exists := runninginstances[endpoint] if exists == false { o := RunningInstance{} tr := &http.Transport{ MaxIdleConns: 10, IdleConnTimeout: 7200 * time.Second, DialContext: (&net.Dialer{ Timeout: 30 * time.Second, KeepAlive: 30 * time.Second, DualStack: true, }).DialContext, } o.client = http.Client{Transport: tr} o.Status = KEEPALIVE runninginstances[endpoint] = o } ri_mutex.Unlock() return o } func UpdateRunner(endpoint string, o RunningInstance) { ri_mutex.Lock() runninginstances[endpoint] = o ri_mutex.Unlock() } func GetNodeInfo(endpoint string, o RunningInstance) RunningInstance { /* Checking order * Mastodon/Pleroma * Um..nothing else yet */ var nodeinfo NodeInfo pleromastodon_nodeinfo_uri := "https://" + endpoint + "/nodeinfo/2.0.json" req, _ := http.NewRequest("GET", pleromastodon_nodeinfo_uri, nil) req.Header.Set("User-Agent", "Tusky") pleromastodon_api_resp, err := DoTries(&o, req) if err != nil { return o } else { defer pleromastodon_api_resp.Body.Close() } if pleromastodon_api_resp.StatusCode == 200 { err = json.NewDecoder(pleromastodon_api_resp.Body).Decode(&nodeinfo) if err == nil { o.Software = nodeinfo.Software.Name o.Version = nodeinfo.Software.Version o.LastRun = time.Now().Format(time.RFC3339) defer pleromastodon_api_resp.Body.Close() return o } } // Check the front page index_uri := "https://" + endpoint + "/" req, _ = http.NewRequest("GET", index_uri, nil) req.Header.Set("User-Agent", "Tusky") resp_index, err := DoTries(&o, req) o.LastRun = time.Now().Format(time.RFC3339) if err != nil { o.Status = UNSUPPORTED_INSTANCE logWarn.Print("Unable to connect to " + endpoint + ", giving up") return o } defer resp_index.Body.Close() indexbin, err := ioutil.ReadAll(resp_index.Body) if err != nil { o.Status = UNSUPPORTED_INSTANCE logWarn.Print("Unable to read index of " + endpoint + ", giving up") return o } indexstr := string(indexbin) if strings.Contains(indexstr, "Pleroma") || strings.Contains(indexstr, "Soapbox") { o.Software = "pleroma" o.Version = "guess" } else if strings.Contains(indexstr, "Mastodon") { o.Software = "mastodon" o.Version = "guess" } else if strings.Contains(indexstr, "Gab") { o.Software = "gab" o.Version = "guess" } return o } func CheckInstance(newinstance string, callerEndpoint string) { if settings.Crawl == true && stringexists(newinstance, settings.Banned) == false { // Skip over this if its the same as the endpoint or empty if newinstance == callerEndpoint || newinstance == "" { return } var err error for attempt := 0; attempt > 5; attempt = attempt + 1 { _, err = net.LookupHost(newinstance) if err != nil { logDebug.Print("Unable to resolve " + newinstance + " attempt ", attempt, "/5. Sleeping for 30 seconds") time.Sleep(time.Second * 30) continue } break } if err != nil { logWarn.Print("Unable to resolve ", newinstance, " after 5 attempts, giving up: ", err) return } // Skip over this if its the same as the endpoint if newinstance == callerEndpoint { return } ri_mutex.Lock() o, exists := runninginstances[newinstance] if exists == false || o.Status == KEEPALIVE { m := RunningInstance{} runninginstances[newinstance] = m go StartInstance(newinstance) } ri_mutex.Unlock() } } func StartInstance(endpoint string) { // Check if exists. If so, get the object. If not, create it o := GetRunner(endpoint) o = GetNodeInfo(endpoint, o) UpdateRunner(endpoint, o) if o.Software == "" { return } if o.Software == "pleroma" { logInfo.Print("Starting " + endpoint + " as " + o.Software + " " + o.Version) o.CaptureType = "Poll" UpdateRunner(endpoint, o) PollMastodonPleroma(endpoint, &o) } else if o.Software == "mastodon" { logInfo.Print("Starting " + endpoint + " as " + o.Software + " " + o.Version) o.CaptureType = "Stream" UpdateRunner(endpoint, o) StreamMastodon(endpoint, &o) } else { logWarn.Print("Unsupported endpoint " + endpoint) } }