This commit is contained in:
Dmitry Zuikov 2024-03-20 17:40:24 +03:00
parent 17e3e076fd
commit d1a387cd01
1 changed files with 179 additions and 132 deletions

311
README.md
View File

@ -1,64 +1,115 @@
- [ABOUT](#about){#toc-about} - [ABOUT](#about)
- [Status update - [Status update 2024-03-20](#status-update-2024-03-20)
2024-03-20](#status-update-2024-03-20){#toc-status-update-2024-03-20} - [Status update 2024-03-17](#status-update-2024-03-17)
- [Status update - [What is it](#what-is-it)
2024-03-17](#status-update-2024-03-17){#toc-status-update-2024-03-17} - [Current status](#current-status)
- [What is it](#what-is-it){#toc-what-is-it} - [HOWTO](#howto)
- [Current status](#current-status){#toc-current-status} - [How to install](#how-to-install)
- [HOWTO](#howto){#toc-howto} - [How to generate peers key?](#how-to-generate-peers-key)
- [How to install](#how-to-install){#toc-how-to-install} - [How to run hbs2-peer](#how-to-run-hbs2-peer)
- [How to generate peer's - [How to configure hbs2-peer](#how-to-configure-hbs2-peer)
key?](#how-to-generate-peers-key){#toc-how-to-generate-peers-key} - [How to create a new own repo](#how-to-create-a-new-own-repo)
- [How to run - [How to launch a peer](#how-to-launch-a-peer)
hbs2-peer](#how-to-run-hbs2-peer){#toc-how-to-run-hbs2-peer} - [How to save an encrypted file
- [How to configure (TBD)](#how-to-save-an-encrypted-file-tbd)
hbs2-peer](#how-to-configure-hbs2-peer){#toc-how-to-configure-hbs2-peer} - [FAQ](#faq)
- [How to create a new own - [Why DVCS are not actually
repo](#how-to-create-a-new-own-repo){#toc-how-to-create-a-new-own-repo} distributed](#why-dvcs-are-not-actually-distributed)
- [How to launch a - [Okay, if centralized services are bad, why are you
peer](#how-to-launch-a-peer){#toc-how-to-launch-a-peer} here?](#okay-if-centralized-services-are-bad-why-are-you-here)
- [How to save an encrypted file - [What platforms are supported
(TBD)](#how-to-save-an-encrypted-file-tbd){#toc-how-to-save-an-encrypted-file-tbd} yet?](#what-platforms-are-supported-yet)
- [FAQ](#faq){#toc-faq} - [What is a “reflog”](#what-is-a-reflog)
- [Why DVCS are not actually - [What is the fixme?](#what-is-the-fixme)
distributed](#why-dvcs-are-not-actually-distributed){#toc-why-dvcs-are-not-actually-distributed} - [Contact](#contact)
- [Okay, if centralized services are bad, why are you - [Download](#download)
here?](#okay-if-centralized-services-are-bad-why-are-you-here){#toc-okay-if-centralized-services-are-bad-why-are-you-here} - [Support](#support)
- [What platforms are supported
yet?](#what-platforms-are-supported-yet){#toc-what-platforms-are-supported-yet}
- [What is a "reflog"](#what-is-a-reflog){#toc-what-is-a-reflog}
- [What is the fixme?](#what-is-the-fixme){#toc-what-is-the-fixme}
- [Contact](#contact){#toc-contact}
- [Download](#download){#toc-download}
- [Support](#support){#toc-support}
- [ABOUT](#about) - <a href="#about" id="toc-about">ABOUT</a>
- [What is it](#what-is-it)
- [Current status](#current-status) - <a href="#status-update-2024-03-20"
- [HOWTO](#howto) id="toc-status-update-2024-03-20">Status update 2024-03-20</a>
- [How to install](#how-to-install) - <a href="#status-update-2024-03-17"
- [How to generate peer\'s key?](#how-to-generate-peers-key) id="toc-status-update-2024-03-17">Status update 2024-03-17</a>
- [How to run hbs2-peer](#how-to-run-hbs2-peer) - <a href="#what-is-it" id="toc-what-is-it">What is it</a>
- [How to configure hbs2-peer](#how-to-configure-hbs2-peer) - <a href="#current-status" id="toc-current-status">Current status</a>
- [How to create a new own repo](#how-to-create-a-new-own-repo)
- [How to make a pull request](#how-to-make-a-pull-request) - <a href="#howto" id="toc-howto">HOWTO</a>
- [How to launch a peer](#how-to-launch-a-peer)
- [How to save an encrypted file - <a href="#how-to-install" id="toc-how-to-install">How to install</a>
(TBD)](#how-to-save-an-encrypted-file-tbd) - <a href="#how-to-generate-peers-key"
- [FAQ](#faq) id="toc-how-to-generate-peers-key">How to generate peers key?</a>
- [Why DVCS are not actually - <a href="#how-to-run-hbs2-peer" id="toc-how-to-run-hbs2-peer">How to run
distributed](#why-dvcs-are-not-actually-distributed) hbs2-peer</a>
- [Okay, if centralized services are bad, why are you - <a href="#how-to-configure-hbs2-peer"
here?](#okay-if-centralized-services-are-bad-why-are-you-here) id="toc-how-to-configure-hbs2-peer">How to configure hbs2-peer</a>
- [What platforms are supported - <a href="#how-to-create-a-new-own-repo"
yet?](#what-platforms-are-supported-yet) id="toc-how-to-create-a-new-own-repo">How to create a new own repo</a>
- [What is a \"reflog\"](#what-is-a-reflog) - <a href="#how-to-launch-a-peer" id="toc-how-to-launch-a-peer">How to
- [What is the fixme?](#what-is-the-fixme) launch a peer</a>
- [Contact](#contact) - <a href="#how-to-save-an-encrypted-file-tbd"
- [Download](#download) id="toc-how-to-save-an-encrypted-file-tbd">How to save an encrypted file
- [Support](#support) (TBD)</a>
- [Donate](#donate)
- [Other](#other) - <a href="#faq" id="toc-faq">FAQ</a>
- <a href="#why-dvcs-are-not-actually-distributed"
id="toc-why-dvcs-are-not-actually-distributed">Why DVCS are not actually
distributed</a>
- <a href="#okay-if-centralized-services-are-bad-why-are-you-here"
id="toc-okay-if-centralized-services-are-bad-why-are-you-here">Okay, if
centralized services are bad, why are you here?</a>
- <a href="#what-platforms-are-supported-yet"
id="toc-what-platforms-are-supported-yet">What platforms are supported
yet?</a>
- <a href="#what-is-a-reflog" id="toc-what-is-a-reflog">What is a
“reflog”</a>
- <a href="#what-is-the-fixme" id="toc-what-is-the-fixme">What is the
fixme?</a>
- <a href="#contact" id="toc-contact">Contact</a>
- <a href="#download" id="toc-download">Download</a>
- <a href="#support" id="toc-support">Support</a>
- [ABOUT](#about)
- [What is it](#what-is-it)
- [Current status](#current-status)
- [HOWTO](#howto)
- [How to install](#how-to-install)
- [How to generate peer's key?](#how-to-generate-peers-key)
- [How to run hbs2-peer](#how-to-run-hbs2-peer)
- [How to configure hbs2-peer](#how-to-configure-hbs2-peer)
- [How to create a new own repo](#how-to-create-a-new-own-repo)
- [How to make a pull request](#how-to-make-a-pull-request)
- [How to launch a peer](#how-to-launch-a-peer)
- [How to save an encrypted file
(TBD)](#how-to-save-an-encrypted-file-tbd)
- [FAQ](#faq)
- [Why DVCS are not actually
distributed](#why-dvcs-are-not-actually-distributed)
- [Okay, if centralized services are bad, why are you
here?](#okay-if-centralized-services-are-bad-why-are-you-here)
- [What platforms are supported
yet?](#what-platforms-are-supported-yet)
- [What is a "reflog"](#what-is-a-reflog)
- [What is the fixme?](#what-is-the-fixme)
- [Contact](#contact)
- [Download](#download)
- [Support](#support)
- [Donate](#donate)
- [Other](#other)
# ABOUT # ABOUT
@ -93,10 +144,10 @@ Right now TEST branch is lwwrepo. Tag: 0.24.1-rc1
Repository is available on: Repository is available on:
- HBS2 hbs2://BTThPdHKF8XnEq4m6wzbKHKA6geLFK4ydYhBXAqBdHSP - HBS2 hbs2://BTThPdHKF8XnEq4m6wzbKHKA6geLFK4ydYhBXAqBdHSP
- HTTPS - HTTPS
https://git.hbs2.net/BTThPdHKF8XnEq4m6wzbKHKA6geLFK4ydYhBXAqBdHSP https://git.hbs2.net/BTThPdHKF8XnEq4m6wzbKHKA6geLFK4ydYhBXAqBdHSP
- GitHub https://github.com/voidlizard/hbs2.git - GitHub https://github.com/voidlizard/hbs2.git
## What is it ## What is it
@ -107,60 +158,60 @@ It may be used for storing and distributed syncronization of data.
HBS2 is aimed to take care of: HBS2 is aimed to take care of:
- NAT traversing - NAT traversing
- Peer discovery - Peer discovery
- Notification - Notification
- Distribution - Distribution
- Encryption - Encryption
- Validation (hashes checking, signatures checking) - Validation (hashes checking, signatures checking)
- Storing and obtaining data - Storing and obtaining data
In short, you store data in this storage, and all subscribers are In short, you store data in this storage, and all subscribers are
notified of it and receive a copy of the data. notified of it and receive a copy of the data.
It is a middleware for implementing distributed applications that shares It is a middleware for implementing distributed applications that shares
data. Like a distributed git, for example. (What? git is already data. Like a distributed git, for example. (What? git is already
distributed and\... No, it is not. Not really). distributed and... No, it is not. Not really).
The idea of extracting the minimal sufficent set of primitives for The idea of extracting the minimal sufficent set of primitives for
distributed applications and APIs and let the side applications do the distributed applications and APIs and let the side applications do the
rest. rest.
This is not a \"blockchain\", but heavily uses the approaches that This is not a "blockchain", but heavily uses the approaches that
\"blockchains\" brought to the world. "blockchains" brought to the world.
Using this solution you may treat application data as local. HBS2 will Using this solution you may treat application data as local. HBS2 will
syncronize all the data along the crowd of peers. The apps don\'t need syncronize all the data along the crowd of peers. The apps don't need to
to bother where the other peers are located, where the hosts, ssh keys bother where the other peers are located, where the hosts, ssh keys on
on thouse hosts, auth tokens on thouse hosts, etc. They only need to thouse hosts, auth tokens on thouse hosts, etc. They only need to know
know the references and (optionally) have signing/encryption keys that the references and (optionally) have signing/encryption keys that are
are stored locally or distributed (public parts, of course) stored locally or distributed (public parts, of course) automatically
automatically like any other data. like any other data.
What types of applications may be implemented on top of this? What types of applications may be implemented on top of this?
For an instance: For an instance:
- Distributed file sharing (wip) - Distributed file sharing (wip)
- Distributed git (seems working) - Distributed git (seems working)
- Distributed communications, like a chat or a \"channel\" - Distributed communications, like a chat or a "channel"
- Distibuted ledgers with different types of consensus protocols - Distibuted ledgers with different types of consensus protocols (we're
(we\'re trying not to use \"b\" words) trying not to use "b" words)
- Actually, any sort of applications that require data and network - Actually, any sort of applications that require data and network
The whitepaper is in shortlist, watch the updates. The whitepaper is in shortlist, watch the updates.
Why it is *experimental* ? Well, it\'s on a quite early stage and some Why it is *experimental* ? Well, it's on a quite early stage and some
root data structures, protocols or API may change. root data structures, protocols or API may change.
It also have some known issues with performance and might have some It also have some known issues with performance and might have some
stability issues. We\'re working hard to fix them. stability issues. We're working hard to fix them.
## Current status ## Current status
Version 0.24.1-rc. Version 0.24.1-rc.
Means it's mostly working. We're using it about a year. Means its mostly working. Were using it about a year.
Encryption status: works. Encryption status: works.
@ -171,20 +222,20 @@ Encryption for protocols: implemented, turned on:
So right now it is useful for distributing any data. So right now it is useful for distributing any data.
We're using it for our non-public projects. Were using it for our non-public projects.
# HOWTO # HOWTO
## How to install ## How to install
Assuming you know what the Nix and Nix flakes are ( See Assuming you know what the Nix and Nix flakes are ( See
[nixos.org](https://nixos.org) if you don't ) [nixos.org](https://nixos.org) if you dont )
and nix flake support is turned on on your system: and nix flake support is turned on on your system:
nix profile install github:voidlizard/hbs2/master nix profile install github:voidlizard/hbs2/master
It will take time. Patience, we're working on rolling out cachix, that It will take time. Patience, were working on rolling out cachix, that
will allow binary caches for the project. will allow binary caches for the project.
Alternative option: Alternative option:
@ -193,7 +244,7 @@ Alternative option:
--substituters http://nix.hbs2.net:6000 \ --substituters http://nix.hbs2.net:6000 \
--trusted-public-keys git.hbs2.net-1:HYIYU3xWetj0NasmHrxsWQTVzQUjawOE8ejZAW2xUS4= --trusted-public-keys git.hbs2.net-1:HYIYU3xWetj0NasmHrxsWQTVzQUjawOE8ejZAW2xUS4=
## How to generate peer's key? ## How to generate peers key?
hbs2 keyring-new > new-peer-key.key hbs2 keyring-new > new-peer-key.key
@ -272,16 +323,14 @@ Typically hbs2-peer config is located at
1. Create a new keyring 1. Create a new keyring
```{=html}
<!-- --> <!-- -->
```
hbs2 keyring-new > new.key hbs2 keyring-new > new.key
2. Watch it's public key 2. Watch its public key
```{=html}
<!-- --> <!-- -->
```
hbs2 keyring-list new.key hbs2 keyring-list new.key
Example: Example:
@ -291,16 +340,14 @@ Example:
3. Export repo to the new reflog 3. Export repo to the new reflog
```{=html}
<!-- --> <!-- -->
```
git hbs2 export --public --new eq5ZFnB9HQTMTeYasYC3pSZLedcP7Zp2eDkJNdehVVk git hbs2 export --public --new eq5ZFnB9HQTMTeYasYC3pSZLedcP7Zp2eDkJNdehVVk
4. Add git remote and push 4. Add git remote and push
```{=html}
<!-- --> <!-- -->
```
git remote add mynerepo hbs2://eq5ZFnB9HQTMTeYasYC3pSZLedcP7Zp2eDkJNdehVVk git remote add mynerepo hbs2://eq5ZFnB9HQTMTeYasYC3pSZLedcP7Zp2eDkJNdehVVk
git push mynerepo git push mynerepo
@ -329,19 +376,19 @@ Example:
## Why DVCS are not actually distributed ## Why DVCS are not actually distributed
Reason 1. Because they don't have any content distribution mechanism. Reason 1. Because they dont have any content distribution mechanism.
Common practice right now is using centralized services, which are: Common practice right now is using centralized services, which are:
- Censored - Censored
- Faulty - Faulty
- Not transparent and irresponsible (For customers. They are - Not transparent and irresponsible (For customers. They are responsible
responsible as hell for any sort of goverment-alike structures as hell for any sort of goverment-alike structures before they even
before they even asked for something). asked for something).
- Tracking users - Tracking users
- May use their code regardless of license agreement - May use their code regardless of license agreement
- Giving up the network neutrality in a sake of \<skipped\*\> anyone - Giving up the network neutrality in a sake of \<skipped\*\> anyone but
but customers who pay customers who pay
There are registered examples, how one most popular git service droppped There are registered examples, how one most popular git service droppped
repositoties because they contain some words in README file. repositoties because they contain some words in README file.
@ -350,30 +397,30 @@ And banned accounts for visiting the service from wrong IP address.
And data loss in a cloud storage services because they located all And data loss in a cloud storage services because they located all
replicas in a single data centre which was destroyed by the fire or a replicas in a single data centre which was destroyed by the fire or a
canalization breakthrough. They even don't tell you how many replicas do canalization breakthrough. They even dont tell you how many replicas do
they have for your data. Why? Because fuck you, that's why. they have for your data. Why? Because fuck you, thats why.
Setting own hosts/services for dvcs data hosting. Setting own hosts/services for dvcs data hosting.
Yeah, it\'s the way. But they are Yeah, it's the way. But they are
- Obviously centralized - Obviously centralized
and also: and also:
- Domain name system is compromised - Domain name system is compromised
- Certificate system is compromised by so many ways. - Certificate system is compromised by so many ways.
Why? Because they are ruled by commercial companies working in certaing Why? Because they are ruled by commercial companies working in certaing
jurisdictions. jurisdictions.
What else. Sending patches by email. What else. Sending patches by email.
- Looks more like anecdote today (but still used by someone) - Looks more like anecdote today (but still used by someone)
- Email right now is a centralized service with all the consequences - Email right now is a centralized service with all the consequences
(see above) (see above)
Okay, ley\'s bring the overlay network (VPN), place all our hosts and Okay, ley's bring the overlay network (VPN), place all our hosts and
resources there and will use own DNS. resources there and will use own DNS.
Yeap, it will work. But it will cost you. It is acceptable for an Yeap, it will work. But it will cost you. It is acceptable for an
@ -391,16 +438,16 @@ like this easily.
Also they require trackers, that are centralized web resources. Also they require trackers, that are centralized web resources.
Things like Syncthing don\'t scales, in fact event if you will use git Things like Syncthing don't scales, in fact event if you will use git
repo in syncthing dir, you will face file modification conflicts even if repo in syncthing dir, you will face file modification conflicts even if
you use them alone. you use them alone.
So that\'s why HBS2 came to light. Trust me, if I could use some So that's why HBS2 came to light. Trust me, if I could use some
decentralized solution normally for this I\'d never start this project. decentralized solution normally for this I'd never start this project.
## Okay, if centralized services are bad, why are you here? ## Okay, if centralized services are bad, why are you here?
Is's a mirror for the really distributed repository: Iss a mirror for the really distributed repository:
hbs2://BTThPdHKF8XnEq4m6wzbKHKA6geLFK4ydYhBXAqBdHSP hbs2://BTThPdHKF8XnEq4m6wzbKHKA6geLFK4ydYhBXAqBdHSP
@ -408,18 +455,18 @@ hbs2://BTThPdHKF8XnEq4m6wzbKHKA6geLFK4ydYhBXAqBdHSP
So far we were able to run the hbs2-peer on: So far we were able to run the hbs2-peer on:
- NixOS ( x86_64-linux ) - NixOS ( x86_64-linux )
- Windows WSL+Ubuntu - Windows WSL+Ubuntu
- Debian/rasberri-pi (aarch64-linux) - Debian/rasberri-pi (aarch64-linux)
Probably it will work on MacOS - but we need someone to check. Probably it will work on MacOS - but we need someone to check.
## What is a "reflog" ## What is a “reflog”
Reflog is an implementation of a permanent mutable reference. It has a Reflog is an implementation of a permanent mutable reference. It has a
permanent ID that corresponds to a public signing cryptographic key, and permanent ID that corresponds to a public signing cryptographic key, and
the value, that is calculated from the "state", where the state is a set the value, that is calculated from the “state”, where the state is a set
of all "reference update" transactions. of all “reference update” transactions.
Each transaction is cryptographically signed by the sender, for current Each transaction is cryptographically signed by the sender, for current
reflog implementation sender must be an owner of the private key of the reflog implementation sender must be an owner of the private key of the