Please wait...
My budget is zero, of course. I am monumentally frugal *and* I have two kids
in college. That having been said, I could probably go up to about USD$300
over budget as long as it's a modular solution that I could expand later.
I am inspired by ServeTheHome's "Project TinyMiniMicro" [https://tinyurl.com/y7atwz2e] and it really seems like this is the way to build a small and low power cluster.
Again however, most of these machines run on like 19 volts and I really, really, really want to be able to do backup power without a 120 VAC step in between.
As a data center guy I know that I need for high availability:
* Uninterrupted power. This is far easier at 12 volts than at any other level.
* No fewer than three nodes.
* No fewer than two disks per node.
Assuming no less than 64 GB RAM per node, add in the power and networking stuff, we're probably looking at about USD$1000, which is more than I am able to spend. This means we have to come up with an open-ended solution and build it in phases.
I am inspired by ServeTheHome's "Project TinyMiniMicro" [https://tinyurl.com/y7atwz2e] and it really seems like this is the way to build a small and low power cluster.
Again however, most of these machines run on like 19 volts and I really, really, really want to be able to do backup power without a 120 VAC step in between.
As a data center guy I know that I need for high availability:
* Uninterrupted power. This is far easier at 12 volts than at any other level.
* No fewer than three nodes.
* No fewer than two disks per node.
Assuming no less than 64 GB RAM per node, add in the power and networking stuff, we're probably looking at about USD$1000, which is more than I am able to spend. This means we have to come up with an open-ended solution and build it in phases.
Ya that will make zero budget hard
Sat Jan 13 2024 17:42:44 EST from IGnatius T Foobar
* Uninterrupted power. This is far easier at 12 volts than at any other level.
* No fewer than three nodes.
* No fewer than two disks per node.+ 64G ram
Sorry for the high volume of posts; I am putting a lot of thought into this project.
Still looking around eBay (mostly) for TMM nodes, and they are quite plentiful, some even shipping with 16 GB of memory. I feel like that might actually be enough if I get rid of OinkLab (which I will). I also tested the performance of NVMe storage running over NFS at 1 Gbps and it was still quite respectable: 80% to 90% of native throughput, still completely blowing away the performance of spinny disks. Mini ITX machines, on the other hand, seem to be sought after by gamers, so they command a higher price than I am willing to pay.
That leaves the issue of power. It looks like someone has solved this problem too. The marketplace is flush with 12 volt to 19 volt boost converters for under USD$20. This means I could start with the power brick and add my DC bus later. I really want the solution to end up like a telecom rack, which runs at -48VDC and is a constant float voltage across a bank of batteries, but mine would run at 12 volts and use a single battery (or multiple ones in parallel). I could even tie it into the 12 volt output of my generator!
Question for our resident EE: can boost converters be ganged? It could be interesting to install N+1 of them, for example two 30 amp converters instead of one 8 amp converter per node, and float the voltages on both sides.
Tee hee hee. Now I'm envisioning an adorable little shelf on my telecom wall, and a voltmeter...
As long as they run in parallel i dont see why not.
Sat Jan 13 2024 20:19:00 EST from IGnatius T Foobar
Question for our resident EE: can boost converters be ganged? It could be interesting to install N+1 of them, for example two 30 amp converters instead of one 8 amp converter per node, and float the voltages on both sides.
Hey, and if you run 12v you could use solar panels and an old car battery to easily keep it up after the apocalypse..
well, other than the nuclear winter clouds..
As a data center guy I know that I need for high availability:
* Uninterrupted power. This is far easier at 12 volts than at any
other level.
* No fewer than three nodes.
* No fewer than two disks per node.
It is... hard... to achieve that... with a budget of 0 USD.
Realistically speaking, for a "home production" server, it makes more sense to have a production server + a backup server than it makes to have three redundant servers + an egress proxy, if just because the latter is more expensive and still you have single points of failure despite all of your expenses. I mean, how many ISP subscriptions do you have at home? Do you have redundant connections you can switch to when your main ISP crashes?
Question for our resident EE: can boost converters be ganged? It
could be interesting to install N+1 of them, for example two 30 amp
converters instead of one 8 amp converter per node, and float the
voltages on both sides.
We are talking about converters that give DC output, aren't we? Then it should be safe to connect them in paralel to get more Amps with the same tension.
I'd place some overcurrent protection in the line if I was going to do that.
I would assume most would have that now anyway so each node wont be melting the lines, but ya, would not hurt to add an external one i guess.
Wed Jan 17 2024 04:09:07 EST from darknetuserI'd place some overcurrent protection in the line if I was going to do that.
I guess it wouldn't be that big a deal to put one converter on each node,
since the nodes themselves aren't HA.
The utilities in the place I live now are exceptionally reliable. I am about half a mile from the electric substation, upstream of a lot of other customers, so the occasional tree-on-a-line gets fixed quickly. And the central office for my fiber Internet connection is even closer, right at the end of the street.
Internet pretty much never goes out here. Electricity goes out maybe once a year on average, usually coming back in a couple of hours.
I just need enough runtime to get my generator plugged in, and to keep the computers running when I refuel it.
This isn't a bigtime commercial operation. It's an open source project and a community bulletin board.
The utilities in the place I live now are exceptionally reliable. I am about half a mile from the electric substation, upstream of a lot of other customers, so the occasional tree-on-a-line gets fixed quickly. And the central office for my fiber Internet connection is even closer, right at the end of the street.
Internet pretty much never goes out here. Electricity goes out maybe once a year on average, usually coming back in a couple of hours.
I just need enough runtime to get my generator plugged in, and to keep the computers running when I refuel it.
This isn't a bigtime commercial operation. It's an open source project and a community bulletin board.
I'm warming up to darknetuser's idea of just going with two nodes instead
of three. That's essentially what I have now at the data center, a SuperMicro
"twin". That could free up enough budget to get something with an N100 CPU
and 64 GB memory and a Biggie Disk (from Wendy's of course). I'm even seeing
machines like that with native 12 volt power supplies. If I were to stick
with x86-64 CPU, I could theoretically just use my desktop as a cold standby.
N100 has only E-cores so it can consume as little as 6 watts!
I wish I could take some of these new low-end CPU's for a test drive. I want to see how today's low end compares to a ten year old Xeon.
N100 has only E-cores so it can consume as little as 6 watts!
I wish I could take some of these new low-end CPU's for a test drive. I want to see how today's low end compares to a ten year old Xeon.
I might be able to stick a N3350 + 6g RAM on my network.
It was to replace my old print server. but its still sitting on a shelf.
Wed Jan 24 2024 18:47:56 EST from IGnatius T Foobar
I wish I could take some of these new low-end CPU's for a test drive. I want to see how today's low end compares to a ten year old Xeon.
I think I've got a better way. My old laptop is an almost 12 year old i7.
I will benchmark on that and then assume that future i7's are faster. I really do think I'm just constrained for disk throughput.
Ixnay on the N100 idea. Single channel memory and it can't go high enough for my needs, and I discovered that its full name is "Celeron N100". I've used Celery chips before and they suck.
I will benchmark on that and then assume that future i7's are faster. I really do think I'm just constrained for disk throughput.
Ixnay on the N100 idea. Single channel memory and it can't go high enough for my needs, and I discovered that its full name is "Celeron N100". I've used Celery chips before and they suck.
ya the low ed stuff are all really celerons with fancy new names.
Bus bandwidth is always an issue with desktops tho.
Well, it's official. I ran the Citadel load test utility on my 11+ year old laptop. It outperformed my production server by two orders of magnitude.
It looks like anything I get would be faster than what I have now.
Geez! I added an "operations per second" meter to the Citadel load test program,
so it's now also a crude benchmarking utility as well. dev.citadel.org (which
is identical to uncensored.citadel.org) ran at an average of 4 ops/sec. That's
a server-grade dual 8-core Xeon machine with plenty of memory but with 7200
RPM SATA disks. Back at home, the trusty little NanoPi with its ARM chip
and an NVMe M.2 drive -- which honestly is where I do nearly all of my development
now -- averaged out at 186 ops/sec.
Honestly, I'd host everything on ARM boards if I could get them with more than 8 GB of memory.
So it looks like I can't lose as long as I get decent storage. And yes, I've been blathering on about this for way too long, but it's what currently has my attention, so thank you all for following along with me.
Honestly, I'd host everything on ARM boards if I could get them with more than 8 GB of memory.
So it looks like I can't lose as long as I get decent storage. And yes, I've been blathering on about this for way too long, but it's what currently has my attention, so thank you all for following along with me.
The bigger brother to the one i sent you has 16G and Orange Pi now has a RK3588 option with 32G
Sat Jan 27 2024 12:42:16 EST from IGnatius T Foobar
Honestly, I'd host everything on ARM boards if I could get them with more than 8 GB of memory.
Ooooh, that might be an option then :) Especially now that many of these
boards come with 2.5Gbps Ethernet which is perfectly fine for cluster storage
at this scale. And let's be honest: I am FAR more interesting and talented
than Jeff Geerling.
The T6 is its big brother. I have one of them of course, and i got the 16G version. I think 128g eMMC but like the others, its just to boot, then flip over to m.2 so its not really important how big it was. its also a non-S chip, so more I/O ( actually the one you have i got while waiting for the T6 to be finished. After my disaster with the RockPi crap, i wanted something that actually worked and got tired of waiting so while it wasn't what i 'wanted' it was close enough )
Orange Pi, is my 2nd source that i trust. The 5B plus is basically the same thing as the T6, also a non-s chip, but they offer a 32G option, which is nice.
Sat Jan 27 2024 16:59:15 EST from IGnatius T FoobarOoooh, that might be an option then :) Especially now that many of these boards come with 2.5Gbps Ethernet which is perfectly fine for cluster storage at this scale. And let's be honest: I am FAR more interesting and talented than Jeff Geerling.
Oh and most have 2 Ethernet ports so you can separate out the storage traffic from web.. ( can anyone say ceph? :) )
get a bunch of those boards, build a little rack for them. Would be cool.
https://www.thingiverse.com/thing:1936196