Username: Password:

Author Topic: Linux Connection Throttler/SSL proxy for Poloniex  (Read 534 times)

Offline dj31

  • Trust: (0)
  • Rookie
  • *
  • Posts: 6
    • View Profile
Linux Connection Throttler/SSL proxy for Poloniex
« on: June 25, 2017, 02:42:26 AM »
Disclaimer:

This may work for you, and it may not.   It's not a silver bullet, and you will still get 422's.   No 'optimizations' are performed.  All it does is globally limit the number of simultaneous connections to poloniex.    It currently has the polo IP hard coded, so if it changes -- you gotta change it.  It will redirect all local https requests to poloniex.

Requirements:
- 'nix server
- haproxy installed
- ability to edit hosts file
- port 443 available

Step 1:   Setup haproxy

This is a barebones, no logging, no frills config.
Code: [Select]
global
   maxconn 4096
   user haproxy
   group haproxy
   pidfile     /run/haproxy.pid
   daemon

defaults
   mode  http
   option   httplog
   option   dontlognull
   retries  3
   option redispatch
   maxconn  2000
   timeout connect 10000
   timeout server 50000
   timeout client 50000

# Host HA-Proxy web stats on Port 3306 (that will confuse those script kiddies)
listen HAProxy-Statistics
    bind :3306
    mode http
    option httplog
    option httpclose
    stats enable
    stats uri /haproxy?stats
    stats refresh 20s
    stats show-node
    stats show-legends
    stats show-desc Workaround haproxy for SSL
    stats auth admin:nimda

frontend ssl_relay
    bind :443
    mode tcp
    option tcplog
    option socket-stats
    maxconn  5
    default_backend ssl_polo

backend ssl_polo
   mode tcp
   option tcplog
   balance roundrobin
   hash-type consistent
   server x_polo 104.20.12.48:443

-  Change the stats auth <user>:<password> to your liking.
-  If port 3306 isn't available for stats, change it to whatever you like
-  Set maxconn to however many maximum simultaneous connections you want

You can view stats at:   http://<yourserver>:3306/haproxy?stats

Step 2.  Redirect polo requests

Edit your /etc/hosts file and append the following to the end

Code: [Select]
<haproxyIPAddress>   poloniex.com


I'm sure there are some optimizations/tweaks, but it's a starting point.

Offline AlfonseX

  • Trust: (0)
  • Contributor
  • **
  • Posts: 109
  • Trop de chefs, pas assez d'indiens !
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #1 on: June 25, 2017, 08:41:12 AM »
Cool job ;)
I didn't know haproxy, I will give it a try and dig though its configuration.
How do you find Polo IP?

Keep in touch.
If you think I helped you, give me a drink:
in btc: 12aeQSpytxoehCEptQE8tUJVVSAS42LvXo
in eth: 0x02a611f0c15bccdb6fa8e5e4b0692ff6d77852bd

Offline beer-k0in

  • Trust: (0)
  • Contributor
  • **
  • Posts: 140
  • Beer beer beer
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #2 on: June 25, 2017, 09:38:44 AM »
How do you find Polo IP?

Actually polo uses 2 IPs: 104.20.12.48 and 104.20.13.48. (Bittrex and Kraken using 5).

A tool to get IPs for a domain name is nslookup:

Code: [Select]
$ nslookup poloniex.com

Non-authoritative answer:
Name: poloniex.com
Address: 104.20.13.48
Name: poloniex.com
Address: 104.20.12.48


BK
Tools:
Generator-Gunbot
Gunbot Monitor
Support gmon and send a tip: BTC wallet: 1GJCGZPn6okFefrRjPPWU73XgMrctSW1jT

Offline dj31

  • Trust: (0)
  • Rookie
  • *
  • Posts: 6
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #3 on: June 25, 2017, 01:16:57 PM »
Some slight tweaks.

#1 - use localhost instead of adapter's IP
#2 - use both polo servers on the backend.

New Config
Code: [Select]
global
   maxconn 4096
   user haproxy
   group haproxy
   pidfile     /run/haproxy.pid
   daemon

defaults
   mode  http
   option   httplog
   option   dontlognull
   retries  3
   option redispatch
   maxconn  2000
   timeout connect 10000
   timeout server 50000
   timeout client 50000

listen HAProxy-Statistics
    bind :3306
    mode http
    option httplog
    option httpclose
    stats enable
    stats uri /haproxy?stats
    stats refresh 20s
    stats show-node
    stats show-legends
    stats show-desc Workaround haproxy for SSL
    stats auth admin:nimda

frontend ssl_relay
    bind 127.0.0.1:443
    mode tcp
    option tcplog
    option socket-stats
    maxconn  3
    default_backend ssl_polo

backend ssl_polo
   mode tcp
   option tcplog
   balance roundrobin
   hash-type consistent
   server x_polo 104.20.12.48:443
   server x_polo2 104.20.13.48:443

New /etc/hosts entry
Code: [Select]
127.0.0.1       poloniex.com

Offline Seki92

  • Trust: (0)
  • Rookie
  • *
  • Posts: 7
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #4 on: June 25, 2017, 04:07:30 PM »
error

::::2017/06/25 16:59:10   Error Error: connect ECONNREFUSED 127.0.0.1:443

::::2017/06/25 16:59:11   Error Error: connect ECONNREFUSED 127.0.0.1:443

Offline AlfonseX

  • Trust: (0)
  • Contributor
  • **
  • Posts: 109
  • Trop de chefs, pas assez d'indiens !
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #5 on: June 25, 2017, 07:08:10 PM »
Port 443 is not open, do :

Code: [Select]
sudo iptables -A INPUT -p tcp -m tcp --dport 7777 -j ACCEPT
See https://help.ubuntu.com/community/IptablesHowTo
If you think I helped you, give me a drink:
in btc: 12aeQSpytxoehCEptQE8tUJVVSAS42LvXo
in eth: 0x02a611f0c15bccdb6fa8e5e4b0692ff6d77852bd

Offline agis

  • Trust: (0)
  • Rookie
  • *
  • Posts: 16
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #6 on: June 25, 2017, 07:12:05 PM »
error

::::2017/06/25 16:59:10   Error Error: connect ECONNREFUSED 127.0.0.1:443

::::2017/06/25 16:59:11   Error Error: connect ECONNREFUSED 127.0.0.1:443

haproxy is doing what it's supposed to do: accepting 3 simultaneous connections and rejecting the rest (read the conf you just copied and pasted)

Offline AlfonseX

  • Trust: (0)
  • Contributor
  • **
  • Posts: 109
  • Trop de chefs, pas assez d'indiens !
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #7 on: June 25, 2017, 07:34:23 PM »
Finally, I'm a noob in network concern, what is the purpose of that? I know load balancing and so on, but in this case I have difficulties to understand.

Thanks.
If you think I helped you, give me a drink:
in btc: 12aeQSpytxoehCEptQE8tUJVVSAS42LvXo
in eth: 0x02a611f0c15bccdb6fa8e5e4b0692ff6d77852bd

Offline allanster

  • Trust: (0)
  • Novice
  • *
  • Posts: 83
  • 00100100
    • View Profile
Re: Linux Connection Throttler/SSL proxy for Poloniex
« Reply #8 on: July 03, 2017, 02:50:13 AM »
What they're trying to do is the same thing I'm doing in Windows, we're trying to limit the number of connections from Gunbot to Poloniex. I just spent hours the other night testing different angles of attacking this same problem. Registry hacks, 3rd party apps, etc...

I can't speak to what their purpose is but I suppose it's the same as mine, to reduce 422 timeout errors.

I also want to prevent exceeding the 6 api calls per second rule. Problem is, I don't know how many calls are made per connection or session. So even if I limit it to one connection, I may still get 429 error.

May I ask why I see everyone limiting to 3 connections instead of 6? Has someone already figured this out?
more human than human

Offline allanster

  • Trust: (0)
  • Novice
  • *
  • Posts: 83
  • 00100100
    • View Profile
more human than human