Aaaaaaaaand it's up and running. This is DA BOMB. Anyone who wants to run servers at home should get this service.
The router they sent is a Cisco 871. You're supposed to plug one end into your LAN and the other end into your server network, but I password-recovered the router and changed things around so that the outside and inside interfaces appear as different VLANs on the same physical port. That way I was able to just pick up the second VLAN on my computer and bridge it to the virtual machines.
I have a clusterfuck of NATs behind NATS behind NATs at home. It is a naty arrangement.
Naty. Do you get it?
I just signed up. Currently my ISPs Router/Cable Modem has two broadcast SIDs, they go to my Citadel, to a Google Mesh, and to a traditional Netgear Router.
What is one more router behind the main one and lots of differently poorly configured different Networks by a guy who is really more of a systems admin guy than a networking professional. If I were a Network Pro, work would probably be easier to find. I picked the wrong career path.
Anyhow... We'll see if I can figure this out. Going to have to figure out who my Registrar is again. :)
That's implemented on the customer router, though. The very first thing I did was a password recovery on the router so I could make changes to the configuration :) I saw where they did the IPv6 block and took care of it myself instead of asking for support to do it.
The service is designed to be simple for the customer, though. You plug the router's FastEthernet4 (WAN) port into any access network, and it presents the tunneled subnet on the built in four port switch (FastEthernet0-3). Plug and play, just connect up and configure the assigned addresses on your servers. Just remember however, that when you connect to your own servers from your own network, it's going to hairpin that connection out to the Ace data center unless you've built some sort of short circuit into your local network.
Once I recovered the password on my router, I made some changes.
For starters, I assigned a static IP address to the WAN port, and opened telnet access to it from my local network, so I can connect to it locally and do whatever else I want to. They actually configure a second tunnel just for managing the router, but I disabled that because I don't want anyone inside my network, even cool people like them.
Then I moved the tunneled network. Instead of appearing on Fa0 through Fa3 (the 4-port switch) it now appears on Fa4.72 (VLAN 72 on the WAN port). So now I can put the router anywhere on the network, it only consumes one port, and I didn't need to install a second Ethernet interface into my server. All I had to do was create a second bridge group (br1) and attach it to the existing network interface with VLAN 72 tagged.
Each of my virtual machines now has two virtual network interfaces, one on br0 (the local network) and one on br1 (the tunneled network). This was a totally optional step, and yes it does mean that if someone broke into one of my servers they could access my home network, but who cares? What are they going to do, send spam to my smart televisions?
ProTip: to access your tunneled network without hairpinning through Ace and without modifying anything else, you can put a static route on your desktop computer.
Wed Nov 11 2020 11:21:23 EST from IGnatius T Foobar @ Uncensored
The very first thing I did was a password recovery on the router so I could make changes to the configuration :) I saw where they did the IPv6 block and took care of it myself instead of asking for support to do it.
The service is designed to be simple for the customer, though... ...Just remember however, that when you connect to your own servers from your own network, it's going to hairpin that connection out to the Ace data center unless you've built some sort of short circuit into your local network.
For starters, I assigned a static IP address to the WAN port, and opened telnet access to it from my local network, so I can connect to it locally and do whatever else I want to. They actually configure a second tunnel just for managing the router, but I disabled that because I don't want anyone inside my network, even cool people like them.
Then I moved the tunneled network. Instead of appearing on Fa0 through Fa3 (the 4-port switch) it now appears on Fa4.72 (VLAN 72 on the WAN port). So now I can put the router anywhere on the network, it only consumes one port, and I didn't need to install a second Ethernet interface into my server. All I had to do was create a second bridge group (br1) and attach it to the existing network interface with VLAN 72 tagged.
Each of my virtual machines now has two virtual network interfaces, one on br0 (the local network) and one on br1 (the tunneled network). This was a totally optional step, and yes it does mean that if someone broke into one of my servers they could access my home network, but who cares? What are they going to do, send spam to my smart televisions?
ProTip: to access your tunneled network without hairpinning through Ace and without modifying anything else, you can put a static route on your desktop computer.
Yeah, you're a bit beyond my pay grade at networking. I'd multihome with a NIC on my internal network to achieve what you did - but, with the Pi, I suppose that would require a USB Ethernet or WiFi dongle installed and configured - and for all I know that would require a custom recompile on Raspbian in order to enable. I hadn't considered that maintenance via Telnet will be going out to Ace and then back in across the public network. That is less than ideal. I guess I could just Telnet through my VPN - but then my exit from my VPN service back to my server would still be cleartext over the public network, right? I guess the static route on my desktop is really the way to go. Thanks for that tip.
I don't mind them being able to get into this network - it is just going to be the Citadel, anyhow - which is already a public faced server. I'm comfortable with pretty advanced networking concepts, but I'm not a networking pro by any stretch of the imagination - so it is probably best that I leave their ability to provide me support unrestricted. :D
Should have it and be up and running by this weekend.
I'm sure having that extra range of IP addresses will make me want to come up with other things to use them on, eventually. :)
I only needed three (www, uncensored, dev) so I had them set the reverse DNS on the other two to "stage" and "test" figuring I could use those names for anything I might want to tinker with in the future.
So yeah, to get to your own servers without hairpinning, you've basically got three choices:
1. Dual-home the servers on both the LAN and the server network (not ideal with a Pi)
2. Set a static route on your desktop for the server network, point it to the WAN IP address of the VPN router.
3. Dual-home your desktop, consuming one of your five server network addresses to use it as a jump box. (Careful about this one: your desktop will probably pick up an IPv6 address from the server network and send all your IPv6 traffic out through the tunnel.)
The static route is probably the way to go in your case. No side effects. You just have to remember to set it up on any machine you want to use for that purpose.
Wed Nov 11 2020 15:50:17 EST from IGnatius T Foobar @ Uncensored
The static route is probably the way to go in your case. No side effects. You just have to remember to set it up on any machine you want to use for that purpose.
I guess the only problem with this is that when I want to connect to the BBS, I'll be going all the way out and back in - I'm not even sure how to set a static route on an Amiga, which I'm tricking into thinking has a PPP connection when it is really an ethernet connection over a parallel port...
But, it'll give me a better gauge of what my end users are experiencing. That has been the problem so far... I can't actually connect from inside to the OUTSIDE. I have to connect to the internal IP address of the machine.
Thus I find myself thinking it is up, when it isn't reachable from outside.
As for the static route, it doesn't go "all the way out". It doesn't bypass the router, but it does bypass the tunnel. For example, here's my setup.
My block of tunneled IP addresses is 72.0.224.88/29. My router is attached to the home network at 192.168.1.89. So the static route on my desktop is:
route add -net 72.0.224.88 netmask 255.255.255.248 gw 192.168.1.89
(Adjust accordingly if you're using a different OS, or if I got the syntax wrong like I usually do.)
When I access a server on the 72.0.224.88 network, it is an ordinary routed IP path from my desktop to the router and then to the server. It does NOT hairpin through the Ace data center, it does NOT count towards the 5 Mbps bandwidth cap, and it does NOT leave the local network.
Another option, if your main router has the ability to have multiple inside networks, would be to build a dedicated transport network between the main router and the Ace router, and then put the static route on the main router.
Then everything would "just work" without static routes on your desktop machines.
Thu Nov 12 2020 10:34:01 EST from IGnatius T Foobar @ UncensoredMonitoring is a separate problem to solve. If you want to monitor your servers, you could always install a monitoring app on your smartphone.
When I access a server on the 72.0.224.88 network, it is an ordinary routed IP path from my desktop to the router and then to the server. It does NOT hairpin through the Ace data center, it does NOT count towards the 5 Mbps bandwidth cap, and it does NOT leave the local network.
This is the important part, specifically, "does NOT leave the local network."
It is the hops between my network to Ace, then back from Ace to my public ID, that worry me. The folks between us.
As for monitoring - what I mean is that right now, I'll be hitting the BBS fine, from inside, and I won't realize it is no longer accessible from the outside due to a DHCP renewal. I know I could fix that by having something detect a DHCP renewal and updating my DDNS entry automatically - I've just never got around to it.
Lazy network monitoring.. If I'm coming in from outside, and I can't get in, I know something is wrong. Right now, I hit an internal IP address, and just because I can get in, doesn't mean anyone else can.
If there is a crash, Citadel notifies me about it the next time I log in. I've yet to have a hard crash that didn't recover automatically.
*knocks on silicon*
It is the hops between my network to Ace, then back from Ace to my
public ID, that worry me. The folks between us.
Right. And if you are concerned about the privacy of that part of the link, you should be looking at it, because they freely point out that the tunnel is *not* encrypted. Privacy/security is not the focus of this service.
However, if you set up the static route (72.0.224.96/29 via the IP address of the WAN side of your router), that traffic will *not* hairpin through the tunnel. It will simply be routed with normal forwarding.
Here's how you confirm it:
1. From your desktop, get a continuous ping going to one of your servers.
2. Add the static route.
3. Observe that when the static route is added, your ping time suddenly drops to <1ms
If you are not in New York City, the difference should be dramatic. Even where I am, in the northern suburbs, the difference is easily observable.
Tue Nov 17 2020 12:05:53 EST from IGnatius T Foobar @ UncensoredIt is the hops between my network to Ace, then back from Ace to my
public ID, that worry me. The folks between us.
Right. And if you are concerned about the privacy of that part of the link, you should be looking at it, because they freely point out that the tunnel is *not* encrypted. Privacy/security is not the focus of this service.
However, if you set up the static route (72.0.224.96/29 via the IP address of the WAN side of your router), that traffic will *not* hairpin through the tunnel. It will simply be routed with normal forwarding.
Here's how you confirm it:
1. From your desktop, get a continuous ping going to one of your servers.
2. Add the static route.
3. Observe that when the static route is added, your ping time suddenly drops to <1ms
If you are not in New York City, the difference should be dramatic. Even where I am, in the northern suburbs, the difference is easily observable.
So... an interesting story about multihomed PCs. When Funlove or Klez32 came down - Intel had advanced warning, and we spent days preparing our externally faced systems in the DC to be ready. We went home that weekend, knowing it was going to hit, confident that we had shut every possible door in to our network, in particular, my group, which was a caged and locked DMZ inside the regular DC - felt very confident that our security was above standard P100 Intel security standards.
That weekend, as soon as it started hitting, our monitoring alarms started paging out to us, and we all drove in, wondering WTF had happened. Once the worm hit, it hit EVERYWHERE.
But when we got there, it was clear it wasn't inside our subnet on the DMZ from external sources, nor from any connections to the regular corporate network, which was ALSO getting hammered.
It turned out to be developers who had brought in rogue hotspots and had left their PCs open. The PC got infected through the rogue hotspot, and the PC had persistent connections with permission to corporate servers *and* to our production secret-op DMZ environments.
Thousands of manhours and hundreds of IT staff in Folsom alone, working all week solely on fixing an issue that a handful of rogue APs allowed to happen.
Pretty sure those devs were no longer employed at Intel by the end of that week.
This is the elegance of your solution, as it isolates the Citadel completely from the internal private network.
The tunnel is capped at 5 Mbps, but my Internet service is 1 Gbps. So...
My nightly backup routine consists of:
1. Create a btrfs snapshot of /var/lib/libvirt/images
2. rsync the snapshot to an offsite host
(this uses the host's 1 Gbps Internet, not the guest's 5 Mbps Internet)
3. Delete the snapshot
On the offsite host, those backups are automatically snapshotted and rotated for a week, also using btrfs.
It's completely seamless, and my entire hosting environment can live wherever the tunnel terminates. If I can ever get the tunnel working in software, I could move the whole site back and forth between the two locations any time I want.
I design data center environments for a living :)
Thu Nov 19 2020 16:16:52 EST from IGnatius T Foobar @ UncensoredOh, it gets even better than that :)
The tunnel is capped at 5 Mbps, but my Internet service is 1 Gbps. So...
My nightly backup routine consists of:
1. Create a btrfs snapshot of /var/lib/libvirt/images
2. rsync the snapshot to an offsite host
(this uses the host's 1 Gbps Internet, not the guest's 5 Mbps Internet)
3. Delete the snapshot
On the offsite host, those backups are automatically snapshotted and rotated for a week, also using btrfs.
It's completely seamless, and my entire hosting environment can live wherever the tunnel terminates. If I can ever get the tunnel working in software, I could move the whole site back and forth between the two locations any time I want.
I design data center environments for a living :)
I figured you were in the industry. Elegant solution. I had an opportunity about 2011 to switch my focus to Linux - but then I got caught up in a difficult move, my kid working through Jr. High and High School, and my wive moving up to high level management in national companies. My professional focus got back-burnered - and at this point, a 50 year old white guy who hasn't been cutting edge for 10 years and wants a large salary in the IT field just doesn't seem worth fighting for. :)
I just bought 5 32gb SD cards. My backup solution is going to be taking the SD out of the Pi, making an image of it, burning it to a new SD, putting the new SD back in the Pi, and putting the old one away in storage. I figure a rotation of 5 monthly images is enough for the mission critical purposes of a BBS that is getting maybe 3 to 5 callers a week. :)
I hate backups.
Hooray!
After a number of failed attempts, I FINALLY got Verizon to let me cancel my TV service and keep Internet.
Verizon FiOS is the single finest residential Internet service in the nation, hands down, no contest. As with many people, however, I just don't want the massive pipeline of raw sewage (cable television) pumping into my home. A year ago, I tried to change it up, and ended up with a lower bill, faster Internet, and keeping the TV service. A month ago, I tried again, and they scuttled the attempt but I was still able to return my set top box and cancel DVR service, which pocketed about $25/month.
Today ... while wrapping not-Christmas presents, I figured I'd put on my headset and wait on hold while trying again. And to my surprise, I got a service rep who only asked me once if I reeeeeeeally wanted to cancel TV, after which she gave me what I wanted. I still have the landline phone, which is weird, but getting rid of that would have been $10 *more*. Bundle discount or something, I dunno.
At long last, I am officially a cord cutter!
We have metronet here. ( a mid-west thing i think ) and they have served us quite well.
Only gripe is they didnt tell you upfront that its a "neighborhood NAT" ( i forget the actual term ) and you had to pay extra for a static IP to get incoming. Had to call to ask what the deal was, as it was nowhere on their pages as a service to buy or even an explanation of the default NAT. "Hi, it appears i'm on a NAT or something, can this be fixed? " "oh, everyone does this and you need to order... " That's great.. just could have told me upfront and saved me some headaches and loss of incoming services for a week. ( and no, not everyone NATs entire neighborhoods together.. but whatever )
Thu Dec 24 2020 15:31:54 EST from IGnatius T Foobar
Hooray!
After a number of failed attempts, I FINALLY got Verizon to let me cancel my TV service and keep Internet.
Verizon FiOS is the single finest residential Internet service in the nation, hands down, no contest. As with many people, however, I just don't want the massive pipeline of raw sewage (cable television) pumping into my home. A year ago, I tried to change it up, and ended up with a lower bill, faster Internet, and keeping the TV service. A month ago, I tried again, and they scuttled the attempt but I was still able to return my set top box and cancel DVR service, which pocketed about $25/month.
Today ... while wrapping not-Christmas presents, I figured I'd put on my headset and wait on hold while trying again. And to my surprise, I got a service rep who only asked me once if I reeeeeeeally wanted to cancel TV, after which she gave me what I wanted. I still have the landline phone, which is weird, but getting rid of that would have been $10 *more*. Bundle discount or something, I dunno.
At long last, I am officially a cord cutter!
RFC6598 specifies that range for carrier grade NAT, so that it doesn't conflict with private or public networks. That still doesn't solve your problem, of course, but I suspect a lot of access services will move to that as IPv4 scarcity increases. I think I mentioned in another room that my wireless connection with T-Mobile is IPv6 native and IPv4 CGN, and I've never really noticed a problem. (Of course, I don't run servers on my phone, so there's that.)
FiOS still gives me a globally routable IPv4 address, but they're still not giving me an IPv6 address. :(
So let's talk turkey about small routers! Now that I don't have to support a MoCA network to keep the set top box running, I can switch to any router I want. Not quite sure what I want. Any suggestions? My requirements are:
1. Must support 1 Gbps speeds
2. Not ridiculously expensive
3. Must be able to reload with pfSense, OpenWRT, etc.
I *don't* need wifi and would prefer not to have it in the router. My wifi is handled elsewhere.
I honesty dont remember the internal addresses. Once i realized it was a double NAT, i stopped caring and made the phone call.
If i remember fight the parent company of pfsense does sell some inexpensive ( and stupid expensive ) hardware too. When it finally died i wimped out and just got a commodity netgear, and retired the old pc running pfsense. I had an older mini-dell with room for one network card, + its built in one. forget the model at this point. If i *had* to do it again, most likely id just add another network card to the bigger PVE server ( others dont have room ) and run the router as a VM. I know that is 'eggs in one basket', but its just my house. Being down a few hours while i scramble to put some hardware up isn't a huge deal.
2020-12-26 12:35 from IGnatius T Foobar
Does the neighborhood NAT distribute addresses in the 100.64.0.0/10
range?
RFC6598 specifies that range for carrier grade NAT, so that it doesn't
conflict with private or public networks. That still doesn't solve
your problem, of course, but I suspect a lot of access services will
move to that as IPv4 scarcity increases. I think I mentioned in
another room that my wireless connection with T-Mobile is IPv6 native
and IPv4 CGN, and I've never really noticed a problem. (Of course, I
don't run servers on my phone, so there's that.)
FiOS still gives me a globally routable IPv4 address, but they're
still not giving me an IPv6address. :(
So let's talk turkey about small routers! Now that I don't have to
support a MoCA network to keep the set top box running, I can switch to
any router I want. Not qquite sure what I want. Any suggestions? My
requirements are:
1. Must support 1 Gbps speeds
2. Not ridiculously expensive
3. Must be able to reload with pfSense, OpenWRT, etc.
I *don't* need wifi and would prefer not to have it in the router. My
wifi is handled elsewhere.
Pretty much any Mikrotik RouterBoard would do.
Their low end stuff is consumer hardware running enterprise software. It is quite affordable. And AFAIK you can reflash many of the models with OpenWTRT.
They provide block diagrams and all the goodies a telecomm engineer needs because they build for internet services providers. Stock firmware can do fine grained packet filtering, arp management, professional quality of service, you name it.
The devices that have separate switches often support cpu-offloading -> you offload the network logic from the CPU into each switch chip. If you are that anal for performance.
You can also tweak MTUs and run a variety of scripts, cronjobs and services if need be. For a home network, it means you can run ad filters if you are resourceful.
Biggest drawback is that the documentation in English sort of sucks.
For the price these things are the next best thing to getting an ALIX or a Soekris and slapping a router operating system on them.
Suffices to say I got a bunch of mines with the idea or replacing the firmware by openvrt, and in the end of the day I let the stock firmware stay because it just rocks for small office / network.